{"id":4403,"date":"2021-04-11T12:00:18","date_gmt":"2021-04-11T12:00:18","guid":{"rendered":"https:\/\/conversion.com\/?p=4403"},"modified":"2024-08-20T10:45:50","modified_gmt":"2024-08-20T10:45:50","slug":"increasing-experiment-velocity-part-2","status":"publish","type":"post","link":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/","title":{"rendered":"Going from 10 to 100 experiments per year: Moving ideas through the pipe"},"content":{"rendered":"<section class=\"c-post-content\" data-ref=\"case-content\">\n\t<div class=\"post-content\">\n\t\t<div class=\"post-content__container container container--medium\">\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--links\">\n\t\t\t\t<div class=\"sticky-menu\" data-ref=\"case-content-menu\">\n\t\t\t\t\t<h3 class=\"sticky-menu__title\">Contents<\/h3>\n\t\t\t\t\t<ul class=\"sticky-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link  active\" href=\"#plumbinghowtomoveexperimentsideasthroughthepipe\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Plumbing \u2013 How to move experiments &#038; ideas through the pipe<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#standardization\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Standardization<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#intake\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Intake<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#prioritization\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Prioritization<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#software\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Software<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#workflows\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Workflows<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t<\/div>\n\t\t\t<\/aside>\n\n\t\t\t<div class=\"post-content__content\">\n\t\t\t\t<ul class=\"post-content__sections\">\n\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"plumbinghowtomoveexperimentsideasthroughthepipe\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Part 2: Plumbing \u2013 How to move experiments &#038; ideas through the pipe<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p><strong>Note:<\/strong>\u00a0This article is part II in a series dedicated to helping you increase your online experiment<br \/>\nvelocity. Stay tuned for future instalments.<\/p>\n<p>Part 1:\u00a0<a href=\"http:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-1\/\">Building the Frame<\/a><\/p>\n<p>Plumbing gets sh*t from point A to point B. In the world of experimentation, plumbing is all about what is going on behind the scenes to make an experiment exist. Marketers and software providers often talk about flashy results and high velocity programs while omitting the blood, sweat, tears, and innovation that goes into achieving those results. It\u2019s not an easy task.<\/p>\n<p>Behind the scenes there are real people following processes and protocols that achieve these outcomes. A good plumbing system should allow a great idea to flow from anywhere in your organization back into your bank account without getting stifled or politicized along the way. During that journey the idea will get shaped, refined, supported, designed, communicated, developed, QA\u2019d, launched, analyzed, deployed, etc. The result that took you ten minutes to read, often took tens of hours to produce.<\/p>\n<p><strong>Production can be expensive.<\/strong><\/p>\n<p>The costs of running an experimentation program typically fall into two major categories:<\/p>\n<ol>\n<li>Human Capital<\/li>\n<li>Technology &amp; Infrastructure.<\/li>\n<\/ol>\n<p>The technology costs usually scale with traffic volumes and impressions, while the human capital costs scales to match experiment velocity (you need bodies to design, develop, and analyze experiments). Because of this, as your output increases your human resources costs will become a proportionally larger part of your program.<\/p>\n<p>Similar to optimizing your customer experience, you can see the value of optimizing your process. This post should act as a guide, highlighting the key decision points and techniques that are part of crafting an effective and efficient experimentation process (the plumbing) in your organization.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__index\">\n\t\t\t\t\t\t\t\t<h3 class=\"post-content__index-title\">Contents<\/h3>\n\t\t\t\t\t\t\t\t<ul class=\"post-content__index-links\" data-ref=\"case-content-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#plumbinghowtomoveexperimentsideasthroughthepipe\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tPlumbing \u2013 How to move experiments &#038; ideas through the pipe\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#standardization\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tStandardization\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#intake\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tIntake\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#prioritization\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tPrioritization\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#software\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tSoftware\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#workflows\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tWorkflows\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"standardization\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Standardization<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Creating an experiment is not a one-person job. There are many people filling a lot of different roles, often across different teams and departments. Each of these teams and departments likely use different technology and have different means of communicating. Do your developers work in the same systems as your marketers? Do they have the same priorities?<\/p>\n<p>This challenge makes compliance with a unified system one of the single most important jobs for the leader of an experimentation program and is the first step in laying down your effective process.<\/p>\n<p>At Conversion, we suggest the introduction of an <strong>Experimentation Guide<\/strong>, that acts as the operating manual for the program. This 8-15 page PDF document should contain all of the\u00a0operational standards\u00a0that will govern experiment related work across the organization. Creating such a document will require leaders from each team involved in testing to agree on how they will operate inside the program.<\/p>\n<p>Common pages of this document should include:<\/p>\n<ul>\n<li>How to write a hypothesis<\/li>\n<li>The process for prioritization<\/li>\n<li>How to conduct user research<\/li>\n<li>Data and validity standards<\/li>\n<li>How to submit an idea<\/li>\n<li><a href=\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-1\/\">The production process from idea to archive<\/a><\/li>\n<li>Naming conventions and tagging<\/li>\n<li>Roles and responsibilities<\/li>\n<li>Key Metrics and KPIs<\/li>\n<li>FAQs<\/li>\n<\/ul>\n<p>This document should be produced not only for education, but also for inspiration. Some experimentation topics, such as statistics, can be intimidating for people who may want to get involved with the program. If written properly it should be easily understood by absolute beginners and should leave them with an understanding of the basics of how experiments are run inside the organization. When complete, the Experimentation Guide should live in a centrally accessible location and be updated periodically as the program grows. New employees should be trained to these standards, and existing team members should be expected to uphold them.<\/p>\n<p>This level of detail is not required when a program is just trying to get off the ground, as your focus should be on showcasing the value of experimentation. As your program grows and begins to involve multiple people and multiple teams with independent goals a degree of standardization and buy-in is required. Note: it is common for different teams to want to follow their own standards. While some degree of variance is acceptable, the important thing is that the results and insights are centrally formatted, stored, and circulated. If teams are resistant, the lack of insight sharing is usually the most compelling reason that will bring stakeholders together.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"intake\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Intake<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Great ideas can come from anywhere. In order to achieve scale with your program, idea generation cannot be a bottleneck. You must find a way to tap into the broader organization and intake ideas from those closest to the problem. An important distinction to make is that there is a major difference between the cultural efforts that cause people to\u00a0<strong><em>want<\/em><\/strong>\u00a0to submit ideas (motivation) and the process improvements that make it so people\u00a0<strong><em>can<\/em><\/strong>\u00a0submit ideas (ease). This article will focus on the latter. Culture is a beast for a future article.<\/p>\n<p>Our ideal outcome is that it is easy for any member of the organization to submit\u00a0<strong>high quality<\/strong>\u00a0ideas.<\/p>\n<p>While we want to have few\u00a0<strong>accessibility barriers<\/strong>\u00a0to submit an idea, we still want to introduce several\u00a0<strong>cognitive barriers\u00a0to entry. <\/strong>What I mean by this is that there has to be some onus put on the submitter to provide adequate thought into the idea and back it up with sufficient evidence. We are trying to avoid the \u201cthrow it over the fence\u201d trap that many organizations fall into. Without a cognitive barrier your intake will become a dumping ground for ideas like \u201cUpdate the new form\u201d, which is essentially meaningless without additional information. The submission of high volume and low quality ideas will create political stress on the program as people will feel like they are submitting ideas that are not being actioned. Ensuring the right level of resistance is important to ensure a flowing list of high quality ideas. Note that over time you may want to adjust the resistance for both cognitive and accessibility barriers. A new program will likely have fewer cognitive barriers, and a more advanced program will likely introduce more. This is where the experimentation guide can come in handy. If individuals buy-in into the idea of standardization and there is easy access to definitions you can demand higher quality idea input from the team.<\/p>\n<p>Tips for a successful intake process:<\/p>\n<ol>\n<li><strong>Centrally accessible<\/strong>: People need to know where to submit ideas. This can be an online form, or existing ticketing system.<\/li>\n<li><strong>Standardized<\/strong>: Require specific fields and force users to come to specific realizations. Common fields like hypothesis, and evidence should be required. Organizations will only learn a finite number of technical terms outside their domain. I recommend sticking to these two simple criteria until you are confident that the organization is ready for more terms to be added to your intake process. Leading with advanced terms like minimum detectable effect (MDE) and statistical significance can be intimidating and turn away potentially great ideas. Provide documentation and instruction for your fields and work your way up to all the fields you want.<\/li>\n<li><strong>Have feedback loops<\/strong>: Intake without feedback is one of the most common mistakes I see. Worst case scenario is that the organization\u00a0<strong><em>feels<\/em><\/strong>\u00a0like they are submitting ideas and not seeing them come to life. Regardless of whether an idea gets developed people\u00a0<strong>need<\/strong>\u00a0to receive feedback. Accepting the idea? Great, tell them where it is slotting in. Need more information? That\u2019s okay, train them to fill out the additional information. Rejecting the idea? No problem, tell the person why. Rejecting ideas can be politically tumultuous if it isn\u2019t turned into an educational opportunity. Coach individuals on their submissions so they can ensure their next idea is impactful and more likely to be accepted. If there are particularly influential individuals submitting their first ideas I recommend working directly with them to sculpt their idea into something that goes through. Their buy-in to the system will be worth more over the long-term than the resources spent to run the experiment.<\/li>\n<li><strong>Educational<\/strong>: Intake is your biggest education opportunity for the broader organization. It\u2019s the time when the submitter\u2019s motivations are aligned with yours. Guide them to think about what they are trying to achieve using your standardized framework and challenge their idea from a few angles.<\/li>\n<\/ol>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"prioritization\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Prioritization<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>It is very common for opportunity cost to be the single biggest constraint of your testing program. You can only produce a finite number of all the great ideas you receive. This means you need to pick and choose which ideas will give your program the best chance of success.<\/p>\n<p>There are three key criteria to consider when determining the order of your backlog. We use\u00a0The PIE Framework at Conversion.<\/p>\n<ol>\n<li><strong>Potential<\/strong>: How positive of an outcome do you expect this test to have? Do you have a lot of evidence to support it?<\/li>\n<li><strong>Importance<\/strong>: Is this idea aligned with core business questions and challenges?<\/li>\n<li><strong>Ease<\/strong>: How easy will it be to produce this test? Both politically and technically.<\/li>\n<\/ol>\n<p>While we primarily use PIE, there are a number of other prioritization frameworks that can help you determine which experiments to chew off first. RICE is another popular alternative amongst product owners. If you decide to use your own, ensure it is as objective as possible.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"software\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Software<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>One of the biggest ways to optimize a process is by choosing the right software to manage the program. Software allows for efficiencies through automation that you just can\u2019t get from generic documents and spreadsheets. In order to move from running 10 experiments to running over 100 experiments per year it is a requirement to use some established software to manage your projects.<\/p>\n<p>Your choice of software is going to depend on the level of specialization you require. You will decide between using a generic project management software, or using a specialized experimentation software.<\/p>\n<p>Generic project management solutions:<\/p>\n<ol>\n<li><a href=\"https:\/\/www.atlassian.com\/software\/jira\">JIRA<\/a><\/li>\n<li><a href=\"https:\/\/trello.com\/\">Trello<\/a><\/li>\n<li><a href=\"https:\/\/asana.com\/\">Asana<\/a><\/li>\n<li>Spreadsheets<\/li>\n<\/ol>\n<p>Generic project management tools have your basic needs covered. You will be able to push projects through your process and assign resources to tasks with ease. In our experience these are sufficient up until a point, but tend to fall apart when trying to automate experimentation specific tasks, analyze\/store robust data, and integrate with experimentation software.<\/p>\n<p>For those ready to consider a more specialized solution, Conversion has developed an experimentation project management software called <a href=\"https:\/\/liftmap.com\/\">Liftmap<\/a>. Built by experimentation experts, Liftmap provides the architecture and blueprint needed to run a high-powered experimentation program.<\/p>\n<p><a href=\"https:\/\/liftmap.com\/\">Liftmap<\/a>\u00a0focuses on providing teams with the following:<\/p>\n<ul>\n<li>An expert process (fully customizable)<\/li>\n<li>A system to intake &amp; prioritize test ideas<\/li>\n<li>Integrations with leading software<\/li>\n<li>A centralize place to communicate<\/li>\n<li>A place to conduct results analyses<\/li>\n<li>A fully searchable repository and tagging system<\/li>\n<li>An overview of program KPIs<\/li>\n<li>Visibility into production roadmap<\/li>\n<li>A system for linking experiments to business objectives<\/li>\n<\/ul>\n<p>After years of being exclusively used by the Conversion team and our clients, <a href=\"https:\/\/liftmap.com\/\">Liftmap is now available to the public<\/a>.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"workflows\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Workflows<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>The nuts and bolts. What should the process actually look like?<\/p>\n<p>While this process will vary slightly from team to team based on available resources and team structure, every experiment should go through similar stages. Most commonly we see programs use the following stages:<\/p>\n<ol>\n<li>Intake<\/li>\n<li>Strategy<\/li>\n<li>Design<\/li>\n<li>Development<\/li>\n<li>QA<\/li>\n<li>Live<\/li>\n<li>Analysis<\/li>\n<li>Implementation<\/li>\n<\/ol>\n<p>Within each of these stages you will need to have \u201cstatuses\u201d as there are many steps that have to occur to complete each of these stages. For example the development stage at Conversion contains over 15 statuses (ex. Briefs, code review, goals, etc.). I recommend reviewing the 8 stages mentioned above with your team and considering what statuses you may need to create within them.<\/p>\n<p>Stages and statuses are valuable because they help provide a lot of clarity for stakeholders. You want to make the system detailed enough that you don\u2019t have to answer the question \u201cwhere is this test at?\u201d, but not too detailed that it\u2019s not clear how to interpret the status as an outsider.<\/p>\n<p>If your organization is able to address all of the sections above you will be well on your way to having an effective plumbing system for your experimentation program. You won\u2019t hear flashy case studies about the behind the scenes but the process really is the backbone of any program. If you think you are doing well, consider the most successful programs in the world have spent years refining their process to the point where many are self-service<br \/>\nand anyone in the organization can launch an experiment. There is always room for improvement!<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t\n\t\t\t\t<\/ul>\n\t\t\t<\/div>\n\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--form\">\n\t\t\t\t<div class=\"sticky-newsletter\" data-target=\"post-newsletter-anchor\" data-device=\"desktop\">\n                \t<div class=\"post-newsletter\" data-target=\"post-newsletter\">\n    <h4 class=\"post-newsletter__title\">Real-world growth experiments. In your inbox. Every week. <\/h4>\n            <div class=\"post-newsletter__form post-newsletter__form--world kam-world\">\n        <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script> <script>   hbspt.forms.create({     region: \"na1\",     portalId: \"9358319\",     formId: \"4e927e03-4f90-466a-8646-7f94947f860c\"   }); <\/script>\n    <\/div>\n            <div class=\"post-newsletter__form post-newsletter__form--uk-us kam-uk-us\">\n            <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script><script>hbspt.forms.create({ region: \"na1\", portalId: \"9358319\", formId: \"03e799e7-ee63-4857-9182-cb3a687dba40\" });<\/script>\n        <\/div>\n        <div class=\"post-newsletter__terms\"><p>Join 5,000+ optimizers who subscribe to our content<\/p>\n<\/div>\n<\/div>\n\t\t\t\t<\/div>\n            <\/aside>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":44,"featured_media":4405,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[12,28],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Going from 10 to 100 experiments per year: Moving ideas through the pipe | Conversion<\/title>\n<meta name=\"description\" content=\"Experiment velocity is important. Learn how to build an experiment, run it, and produce successful results, all within a timely manner.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Going from 10 to 100 experiments per year: Moving ideas through the pipe | Conversion\" \/>\n<meta property=\"og:description\" content=\"Experiment velocity is important. Learn how to build an experiment, run it, and produce successful results, all within a timely manner.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/\" \/>\n<meta property=\"og:site_name\" content=\"Conversion\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-11T12:00:18+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-20T10:45:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/conversion.com\/wp-content\/uploads\/2023\/04\/going-fron-1-to-100-experiments-part-2_1600x1000_COMPRESSED-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1600\" \/>\n\t<meta property=\"og:image:height\" content=\"1000\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Mike St Laurent\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/conversion.com\/wp-content\/uploads\/2023\/04\/going-fron-1-to-100-experiments-part-2_1600x1000_COMPRESSED-1.png\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mike St Laurent\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/\"},\"author\":{\"name\":\"Mike St Laurent\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/af268f8f2dbd6a8eadd8860e089d952d\"},\"headline\":\"Going from 10 to 100 experiments per year: Moving ideas through the pipe\",\"datePublished\":\"2021-04-11T12:00:18+00:00\",\"dateModified\":\"2024-08-20T10:45:50+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/\"},\"wordCount\":11,\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"articleSection\":[\"Conversion Optimization\",\"Strategy\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/\",\"url\":\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/\",\"name\":\"Going from 10 to 100 experiments per year: Moving ideas through the pipe | Conversion\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/#website\"},\"datePublished\":\"2021-04-11T12:00:18+00:00\",\"dateModified\":\"2024-08-20T10:45:50+00:00\",\"description\":\"Experiment velocity is important. Learn how to build an experiment, run it, and produce successful results, all within a timely manner.\",\"breadcrumb\":{\"@id\":\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/conversion.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Going from 10 to 100 experimen&hellip;\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/conversion.com\/#website\",\"url\":\"https:\/\/conversion.com\/\",\"name\":\"Conversion\",\"description\":\"Better results with evidence\",\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/conversion.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/conversion.com\/#organization\",\"name\":\"Conversion\",\"url\":\"https:\/\/conversion.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"contentUrl\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"width\":1821,\"height\":193,\"caption\":\"Conversion\"},\"image\":{\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/af268f8f2dbd6a8eadd8860e089d952d\",\"name\":\"Mike St Laurent\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4b41d95a6d6a428e6af659935dd55a9e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/4b41d95a6d6a428e6af659935dd55a9e?s=96&d=mm&r=g\",\"caption\":\"Mike St Laurent\"},\"url\":\"https:\/\/conversion.com\/blog\/author\/mikestlaurent\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Going from 10 to 100 experiments per year: Moving ideas through the pipe | Conversion","description":"Experiment velocity is important. Learn how to build an experiment, run it, and produce successful results, all within a timely manner.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/","og_locale":"en_GB","og_type":"article","og_title":"Going from 10 to 100 experiments per year: Moving ideas through the pipe | Conversion","og_description":"Experiment velocity is important. Learn how to build an experiment, run it, and produce successful results, all within a timely manner.","og_url":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/","og_site_name":"Conversion","article_published_time":"2021-04-11T12:00:18+00:00","article_modified_time":"2024-08-20T10:45:50+00:00","og_image":[{"width":1600,"height":1000,"url":"https:\/\/conversion.com\/wp-content\/uploads\/2023\/04\/going-fron-1-to-100-experiments-part-2_1600x1000_COMPRESSED-1.png","type":"image\/png"}],"author":"Mike St Laurent","twitter_card":"summary_large_image","twitter_image":"https:\/\/conversion.com\/wp-content\/uploads\/2023\/04\/going-fron-1-to-100-experiments-part-2_1600x1000_COMPRESSED-1.png","twitter_misc":{"Written by":"Mike St Laurent"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/#article","isPartOf":{"@id":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/"},"author":{"name":"Mike St Laurent","@id":"https:\/\/conversion.com\/#\/schema\/person\/af268f8f2dbd6a8eadd8860e089d952d"},"headline":"Going from 10 to 100 experiments per year: Moving ideas through the pipe","datePublished":"2021-04-11T12:00:18+00:00","dateModified":"2024-08-20T10:45:50+00:00","mainEntityOfPage":{"@id":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/"},"wordCount":11,"publisher":{"@id":"https:\/\/conversion.com\/#organization"},"articleSection":["Conversion Optimization","Strategy"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/","url":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/","name":"Going from 10 to 100 experiments per year: Moving ideas through the pipe | Conversion","isPartOf":{"@id":"https:\/\/conversion.com\/#website"},"datePublished":"2021-04-11T12:00:18+00:00","dateModified":"2024-08-20T10:45:50+00:00","description":"Experiment velocity is important. Learn how to build an experiment, run it, and produce successful results, all within a timely manner.","breadcrumb":{"@id":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/conversion.com\/blog\/increasing-experiment-velocity-part-2\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/conversion.com\/"},{"@type":"ListItem","position":2,"name":"Going from 10 to 100 experimen&hellip;"}]},{"@type":"WebSite","@id":"https:\/\/conversion.com\/#website","url":"https:\/\/conversion.com\/","name":"Conversion","description":"Better results with evidence","publisher":{"@id":"https:\/\/conversion.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/conversion.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/conversion.com\/#organization","name":"Conversion","url":"https:\/\/conversion.com\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/","url":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","contentUrl":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","width":1821,"height":193,"caption":"Conversion"},"image":{"@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/conversion.com\/#\/schema\/person\/af268f8f2dbd6a8eadd8860e089d952d","name":"Mike St Laurent","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/4b41d95a6d6a428e6af659935dd55a9e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4b41d95a6d6a428e6af659935dd55a9e?s=96&d=mm&r=g","caption":"Mike St Laurent"},"url":"https:\/\/conversion.com\/blog\/author\/mikestlaurent\/"}]}},"_links":{"self":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/4403"}],"collection":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/users\/44"}],"replies":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/comments?post=4403"}],"version-history":[{"count":5,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/4403\/revisions"}],"predecessor-version":[{"id":7306,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/4403\/revisions\/7306"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media\/4405"}],"wp:attachment":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media?parent=4403"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/categories?post=4403"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}