{"id":4428,"date":"2021-04-11T12:00:54","date_gmt":"2021-04-11T12:00:54","guid":{"rendered":"https:\/\/conversion.com\/?p=4428"},"modified":"2024-08-20T11:34:13","modified_gmt":"2024-08-20T11:34:13","slug":"the-roi-of-experimentation-and-conversion-rate-optimization-cro","status":"publish","type":"post","link":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/","title":{"rendered":"The ROI of Experimentation and Conversion Rate Optimization (CRO)"},"content":{"rendered":"<section class=\"c-post-content\" data-ref=\"case-content\">\n\t<div class=\"post-content\">\n\t\t<div class=\"post-content__container container container--medium\">\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--links\">\n\t\t\t\t<div class=\"sticky-menu\" data-ref=\"case-content-menu\">\n\t\t\t\t\t<h3 class=\"sticky-menu__title\">Contents<\/h3>\n\t\t\t\t\t<ul class=\"sticky-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link  active\" href=\"#\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\"><\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#therightsideofthebet\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">The right side of the bet<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#levers\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Levers<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#howtocalculatetheroiofyourcroprogram\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">How to calculate the ROI of your CRO program<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#benchmarks\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Benchmarks<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#therealbenefitsofexperimentationandcro\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">The real benefits of\u00a0 experimentation and CRO<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t<\/div>\n\t\t\t<\/aside>\n\n\t\t\t<div class=\"post-content__content\">\n\t\t\t\t<ul class=\"post-content__sections\">\n\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>CRO and experimentation ROI is (and likely will always be) one of the most hotly debated and divisive topics in the field. However, companies like Microsoft, TaylorMade, and Spotify know that testing is a highly effective way to develop a product and innovate. Moreover, these companies understand that sustaining a successful practice will positively impact their bottom line.<\/p>\n<p>Thankfully, I have noticed a shift in conversation from asking \u2018why should we invest in experimentation?\u2019 to discussions centred on the best way to communicate the overall value of experimentation to your organization. In this guide, I definitively address common objections with real-world applications that help communicate the ROI behind building your practice.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__index\">\n\t\t\t\t\t\t\t\t<h3 class=\"post-content__index-title\">Contents<\/h3>\n\t\t\t\t\t\t\t\t<ul class=\"post-content__index-links\" data-ref=\"case-content-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#therightsideofthebet\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tThe right side of the bet\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#levers\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tLevers\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#howtocalculatetheroiofyourcroprogram\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tHow to calculate the ROI of your CRO program\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#benchmarks\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tBenchmarks\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#therealbenefitsofexperimentationandcro\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tThe real benefits of\u00a0 experimentation and CRO\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"therightsideofthebet\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">The right side of the bet<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>In the gambling industry, the phrase \u201cthe house always wins\u201d is a popular adage. There\u2019s a very good reason for this. While the casino doesn\u2019t necessarily profit from every game played, the odds are tilted in its favor so that across a large volume of games played, the cash always ends up back in the coffers. Players may walk out of the casino with more or less money than before, but the average person leaves with less.<\/p>\n<p>Experimentation is not all that different from the casino in this case. The majority of companies may not have a winner for every experiment, but with enough tests, the odds are good that you will come out ahead.<\/p>\n<p>In his book \u201cAntiFragile,\u201d author Nassim Taleb introduces the idea of convexity. Without being too tangential, convexity essentially describes a system that benefits from disorder. \u201cDisorder,\u201d in this case, represents an unpredictable input that could cause something to either gain or lose randomly. When a system is convex, it receives more benefit in what it gains than it loses when harmed.<\/p>\n<div class=\"main-content__image\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/08\/10210029\/convexity-diagram.png\" width=\"585\" height=\"324\" \/><\/div>\n<p>So what does this have to do with experimentation and CRO?<\/p>\n<p>The beauty of experimentation is that the downside is capped (you don\u2019t ship a losing experience), but the upside is infinite (there\u2019s no limit on how much you can lift a conversion rate, even &gt;100%).<\/p>\n<p>With an extensive upside and limited downside, CRO is a great example of a convex system. The \u201cdisorder\u201d in this example is every time an A\/B test is introduced to the system. Each A\/B test (called a trial) creates uncertainty, and thanks to convexity, enough trials lead to positive results over time.<\/p>\n<p>The house always wins.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"levers\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Levers<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>One of the most humbling things about experimentation is that it\u2019s impossible to predict the outcome of changes made to the user experience. After all, if this were possible, there would be no need to test.<\/p>\n<p>If we accept that we have no control over any individual test output (the result), we should focus our attention on what we can control\u2014the input, research, process, design, etc. While even the most researched test still can\u2019t guarantee a positive outcome of a specific test, it is possible to increase your chances on average with enough rigour. In baseball, even the best hitters of all-time bat below .500, but the top players prepare before their performance to bat at a higher ratio than other players.<\/p>\n<p>So what can we control?<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"howtocalculatetheroiofyourcroprogram\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">How to calculate the ROI of your CRO program<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>To calculate an ROI , there are four levers in every experimenter\u2019s tool kit.<\/p>\n<div class=\"main-content__image\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/08\/10210409\/experimentation-ROI-levers.jpg\" width=\"606\" height=\"173\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/08\/10210409\/experimentation-ROI-levers.jpg\" data-ll-status=\"loaded\" \/><\/div>\n<p>With these four levers, we can create both a payoff and a cost function.<\/p>\n<p><strong>The payoff equation:<\/strong><\/p>\n<div class=\"main-content__image\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/08\/10210612\/the-payoff-equation.png\" width=\"597\" height=\"83\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/08\/10210612\/the-payoff-equation.png\" data-ll-status=\"loaded\" \/><\/div>\n<p><strong>The cost equation:<\/strong><\/p>\n<div class=\"main-content__image\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/08\/10215404\/cost-equation.png\" width=\"612\" height=\"85\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/08\/10215404\/cost-equation.png\" data-ll-status=\"loaded\" \/><\/div>\n<p>For example, let\u2019s use the following values<\/p>\n<ul>\n<li><b>Velocity:<\/b>\u00a02 experiments\/month<\/li>\n<li><b>Quality:<\/b>\u00a030% win rate<\/li>\n<li><b>Impact:<\/b>\u00a0$25,000<\/li>\n<li><b>Cost:<\/b>\u00a0$5,000<\/li>\n<\/ul>\n<p>This gives us a payoff\u00a0<b>$15,000<\/b>\u00a0(2 * 0.3 * 25,000) and a cost of\u00a0<b>$10,000<\/b>\u00a0(2 * 5,000)<\/p>\n<p>The number that matters when calculating your ROI is looking at $15,000 (Gain) \u2013 $10,000 (Cost), which gives us a Net Gain of $5,000 per experiment.<\/p>\n<p>The velocity, quality, impact, and costs will vary month over month in business, but overall it is clear that your company (the house) comes out ahead. If you walked into a casino and there was a game that gave you on average\u00a0<b>$5,000<\/b>\u00a0per spin you would not want to leave that table.<\/p>\n<p>Note: This model assumes that all levers are independent of each other, which is unlikely to be accurate, allow me to explain in the following sections.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"benchmarks\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Benchmarks<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Let\u2019s take a look at these controls in more detail because realistically, each company has varying degrees of control over each lever.<\/p>\n<p><strong>Velocity<\/strong><\/p>\n<p>Velocity is strictly a function of traffic and conversion rate. Simply put, the more traffic and the higher your existing conversion rate, the more experiments you can run. The increased number of tests can be attributed to the statistical nature of testing. To prove causality (whether a change in behavior was due to your test), we need to test to an acceptable limit of statistical confidence (typically 90-95%). Every website in the world has a limit on how many tests it can support statistically. If you want help with this calculation for your website, then send us an\u00a0<a href=\"https:\/\/conversion.com\/contact-us\/\">email<\/a>\u00a0and we\u2019ll be happy to help you out. If you are testing below this number today, you may have room to increase this lever and\u00a0<b>improve<\/b>\u00a0your payout function, but if you are already at the cap, then you are maybe out of luck in this category.<\/p>\n<p>To give you an idea of scale, most organizations we see are running between 25-50 experiments per year, but the top testing companies in the world (Ex. AirBnB, Booking, Microsoft, Google) are running &gt;10,000\/year.<\/p>\n<p>You read that right\u2026 &gt;10,000<\/p>\n<p><strong>Quality<\/strong><\/p>\n<p>Quality is a category that practitioners have a fair bit of control over. Typically it\u2019s normal to see win rates anywhere between 10% and 40%. If you happen to have win rates above or below this range, this is a good indicator that you are doing something wrong. Too low indicates you may not be spending enough time grasping your customer\u2019s challenges, and too high indicates you may have questionable statistical validity or that you are not testing enough innovative changes. By increasing your understanding of the customer\u2019s problem through analytics, customer research, and psychology, you can improve the quality of your metrics. In addition,\u00a0rigorous processes, effective design of experiments, and\u00a0quality assurance\u00a0(QA) can positively affect win rates.<\/p>\n<p>Conversion has spent years defining our rigorous process that consistently produced win rates fluctuating around the 40% mark for our clients. Some mature testing programs have reported ever-declining quality win rates as they optimize the business with thousands of experiments per year. However, our experience at Conversion has not shown this same decline. In fact, our data illustrates that win rates actually increase over the time as we gain insights about our client\u2019s customers.<\/p>\n<p><strong>Impact<\/strong><\/p>\n<p>Impact is almost exclusively controlled by a company\u2019s size. As a matter of fact, experimentation is increasingly more effective the more revenue a company generates through the properties it tests on. A 2% lift for a small business may generate an uplift worth $150,000 over the course of a year, but that same 2% lift for a large enterprise may be worth &gt;$2,500,000 over the course of a year. Same test, same lift, same effort, much more impactful result.<\/p>\n<p>If you think back to our example, you can quickly imagine that going from $25,000 to $1,500,000 (2*0.3*2,500,000) per winning test will impact your payoff\u00a0<b>function<\/b>\u00a0dramatically. Running an experimentation program is both a responsible and beneficial business decision because while the impact grows exponentially, the cost function stays roughly the same (It doesn\u2019t cost a large organization any more to run a test than a small one). If we think back to convexity, this essentially means that larger organizations are even more \u2018convex\u2019 than smaller organizations. Naturally, this makes experimentation a more critical component of their business strategy.<\/p>\n<p>The industry standard to calculate the\u00a0<b>impact per winning<\/b>\u00a0test is to project the winning variation\u2019s lift out for a one-year duration and then apply a discount to that projection (typically between 10-30%) to account for false positives and other contamination issues that arise naturally when testing. When it comes to financial projections, it is better to be on the conservative side.<\/p>\n<p><strong>Cost<\/strong><\/p>\n<p>While organizations have some degree of control over experiment cost, it can be a bit difficult to calculate. Cost is going to be a blend of the hours (salary) spent creating experiments, the technology license used to conduct the tests, and any consultants or partners who assist in test creation. When considering all these inputs, I would expect each experiment to cost somewhere between $3,000 \u2013 $15,000. This will also vary wildly from test to test based on technical complexity and research that went into the creation of the idea. At Conversion, each test is scored as small, medium, or large and each will have varying internal costs.<\/p>\n<p>When you consider that the cost of an experiment is unlikely to vary more than $10,000, while impact can vary 100\u2019s of thousands of dollars, it\u2019s clear cost is not the key lever that leads to a higher payoff.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"therealbenefitsofexperimentationandcro\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">The real benefits of\u00a0 experimentation and CRO<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Now forget everything I just told you.<\/p>\n<p>The advice noted in this article is predicated on your business being of a minimum size.If your company falls into this criteria experimentation\u00a0<b><u>WILL BE<\/u><\/b>\u00a0an ROI positive strategy. With enough trials (tests) it is nearly impossible to lose money doing experimentation.<\/p>\n<p>Even with such substantial ROI potential, the most valuable part of conducting testing has far greater benefits.<\/p>\n<p>Let\u2019s take ROI off the table, and take a closer look at the real benefits of experimentation.<\/p>\n<p><strong>Ego is the Enemy<\/strong><\/p>\n<p>A company void of ego is freeing. People can do their best work and challenge assumptions when the default answer in your organization is \u201clet\u2019s ask the customer\u201d. Experimentation is a great way to remove bias from key decision makers and democratize voices throughout the organization. The person who gets paid the most doesn\u2019t know more than the customer \u2014 Letting customers vote with their wallets will lead to the creation of better experiences. A person\u2019s rank in a company is not strongly correlated to their ability to predict test outcomes. A good experimenter is not better than others at knowing answers. A good experimenter is better at finding answers. Enabling all ranks of an organization to find answers is an incredible, competitive super-power.<\/p>\n<p><strong>Winning both sides of the bet<\/strong><\/p>\n<p>Traditionally in product development and marketing, the alternative to running an A\/B test is to add features to a central roadmap that get released periodically. At Conversion we like to make the distinction between \u201coptimization\u201d and \u201cvalidation\u201d. Optimization is an improvement to an existing feature that would not be there if not for an optimization practice. Validation is the testing of a new feature that was planned to be released as part of the product roadmap. While optimization is the practice of continuous improvement of existing features, there are still many times where you want to introduce new features as the result of a broader creative vision. In these cases these changes may be part of a broader strategy and may not be ROI or KPI positive changes.<\/p>\n<p>The benefit of applying experimentation to the product release cycle is that you can quantify the negative impact of not shipping features that are detrimental to the user experience. When evaluating the financial value, there is no difference to the business if you optimize an experience to generate $100,000 additional dollars in revenue, or if you didn\u2019t ship a feature that would have cost the business $100,000 in revenue because of a poor user experience. We refer to this as \u2018loss aversion\u2019 \u2013 A good program should be able to quantify both additional uncovered revenue, as well as losses averted.<\/p>\n<p><strong>Creating a Safe Space to Innovate<\/strong><\/p>\n<p>Fear of failing is what prevents a lot of organizations from taking the necessary risks that lead to innovation. Experimentation provides a safe framework to test (and fail) innovative changes while being able to measure effects in detail, limit exposure, and roll-back changes swiftly. The cultural benefits when creating a safe way to try new ideas and fail can be substantial. People will be more willing to voice ideas and take (measured) risks. The main benefit to the executive team is that they can be confident in their decisions\u2014<br \/>\nEvery test result offers a risk-weighted recommendation. With this data in hand, the decision maker is able to assess risk and reward.<\/p>\n<div class=\"main-content__image\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/08\/11071319\/Widerfunnel-Vancouver-Office-Spotlight-Techvibes.jpg\" width=\"569\" height=\"283\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/08\/11071319\/Widerfunnel-Vancouver-Office-Spotlight-Techvibes.jpg\" data-ll-status=\"loaded\" \/><\/div>\n<div><\/div>\n<p><strong>Reduce the Cost of Development<\/strong><\/p>\n<p>This might sound counter-intuitive, but let me explain.<br \/>\nAdding more tests should surely add to development\u2026. right?<br \/>\nHowever, if you are in an agile organization, experimentation should be sitting before your product roadmap. Experimentation is the MVP (minimum viable product) method for determining whether something is worth spending development resources on. The mistake many companies make is that they put all their resources into building perfect, fully functional experiences and then proceed to test them.<\/p>\n<p>The difference is that world-class testing organizations test roll-outs, but they also validate ideas earlier on in the process with MVP experiments, or other pseudo-experiments to gauge whether changes warrant continued effort from the development team. A good testing practice ensures engineering is only working on critical and impactful features for users. Nothing will kill development team morale faster than spending months on an experience only to have that work disappear due to a bad test result. Fail early, and fail cheaply.<\/p>\n<p><strong>A Megaphone for the Customer<\/strong><\/p>\n<p>These days, many organizations call themselves \u201ccustomer-centric,\u201d but few put this into practice. Enabling a culture of testing is one of the easiest and more systematic ways to become customer-centric. Give your customers and prospects a seat at your executive table by running decisions by them. A well-structured series of experiments can reveal incredibly rich information about the preferences of your core customer\u2014You just have to ask.<\/p>\n<p><strong>CRO adds compounding value<\/strong><\/p>\n<p>With traditional marketing campaign spend, your organization loses value when you turn off your spend. With experimentation, every winner that you uncover adds to an increasingly better user experience. Each time you uncover a 10% winner, that benefit doesn\u2019t go away when the test is over. You may see the impact diminish over time as the competition adjusts, but the beauty of experimenting is that you can stack these winners on top of one another to drive significant value over time. CRO acts much like a capital investment where you pay for it upfront but you receive an ongoing dividend quarter over quarter.<\/p>\n<p>Finally, assess if you can afford to continue:<\/p>\n<ul>\n<li>Allowing gut decisions to drive your roadmap?<\/li>\n<li>Assigning development resources to underperforming features?<\/li>\n<li>Not learning from customer behaviour?<\/li>\n<li>Stifling junior-level employees\u2019 ideas and contributions?<\/li>\n<li>Putting your business at risk by not testing various user-experience changes?<\/li>\n<\/ul>\n<p>The next time you are asked about the ROI of testing, ask yourself, \u201cwhat is the cost of not testing?\u201d<\/p>\n<p>Becoming an exprimentation-driven company will improve your ability to innovate, grow, measure, learn, brainstorm, and become customer-centric. The cost-benefit analysis goes far beyond ROI and will impact the entire culture of your organization when you commit to the process.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t\n\t\t\t\t<\/ul>\n\t\t\t<\/div>\n\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--form\">\n\t\t\t\t<div class=\"sticky-newsletter\" data-target=\"post-newsletter-anchor\" data-device=\"desktop\">\n                \t<div class=\"post-newsletter\" data-target=\"post-newsletter\">\n    <h4 class=\"post-newsletter__title\">Real-world growth experiments. In your inbox. Every week. <\/h4>\n            <div class=\"post-newsletter__form post-newsletter__form--world kam-world\">\n        <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script> <script>   hbspt.forms.create({     region: \"na1\",     portalId: \"9358319\",     formId: \"4e927e03-4f90-466a-8646-7f94947f860c\"   }); <\/script>\n    <\/div>\n            <div class=\"post-newsletter__form post-newsletter__form--uk-us kam-uk-us\">\n            <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script><script>hbspt.forms.create({ region: \"na1\", portalId: \"9358319\", formId: \"03e799e7-ee63-4857-9182-cb3a687dba40\" });<\/script>\n        <\/div>\n        <div class=\"post-newsletter__terms\"><p>Join 5,000+ optimizers who subscribe to our content<\/p>\n<\/div>\n<\/div>\n\t\t\t\t<\/div>\n            <\/aside>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":44,"featured_media":4429,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[12,28],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>The ROI of Experimentation and Conversion Rate Optimization (CRO) | Conversion<\/title>\n<meta name=\"description\" content=\"Becoming an experimentation-driven company will improve your ability to innovate, grow, measure, learn, and become customer-centric.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The ROI of Experimentation and Conversion Rate Optimization (CRO) | Conversion\" \/>\n<meta property=\"og:description\" content=\"Becoming an experimentation-driven company will improve your ability to innovate, grow, measure, learn, and become customer-centric.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/\" \/>\n<meta property=\"og:site_name\" content=\"Conversion\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-11T12:00:54+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-20T11:34:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/conversion.com\/wp-content\/uploads\/2023\/04\/ROI-of-experimentation_blog_1600x1000_COMPRESSED-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1600\" \/>\n\t<meta property=\"og:image:height\" content=\"1000\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Mike St Laurent\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/conversion.com\/wp-content\/uploads\/2023\/04\/ROI-of-experimentation_blog_1600x1000_COMPRESSED-1.png\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mike St Laurent\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/\"},\"author\":{\"name\":\"Mike St Laurent\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/af268f8f2dbd6a8eadd8860e089d952d\"},\"headline\":\"The ROI of Experimentation and Conversion Rate Optimization (CRO)\",\"datePublished\":\"2021-04-11T12:00:54+00:00\",\"dateModified\":\"2024-08-20T11:34:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/\"},\"wordCount\":9,\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"articleSection\":[\"Conversion Optimization\",\"Strategy\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/\",\"url\":\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/\",\"name\":\"The ROI of Experimentation and Conversion Rate Optimization (CRO) | Conversion\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/#website\"},\"datePublished\":\"2021-04-11T12:00:54+00:00\",\"dateModified\":\"2024-08-20T11:34:13+00:00\",\"description\":\"Becoming an experimentation-driven company will improve your ability to innovate, grow, measure, learn, and become customer-centric.\",\"breadcrumb\":{\"@id\":\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/conversion.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The ROI of Experimentation and&hellip;\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/conversion.com\/#website\",\"url\":\"https:\/\/conversion.com\/\",\"name\":\"Conversion\",\"description\":\"Better results with evidence\",\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/conversion.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/conversion.com\/#organization\",\"name\":\"Conversion\",\"url\":\"https:\/\/conversion.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"contentUrl\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"width\":1821,\"height\":193,\"caption\":\"Conversion\"},\"image\":{\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/af268f8f2dbd6a8eadd8860e089d952d\",\"name\":\"Mike St Laurent\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4b41d95a6d6a428e6af659935dd55a9e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/4b41d95a6d6a428e6af659935dd55a9e?s=96&d=mm&r=g\",\"caption\":\"Mike St Laurent\"},\"url\":\"https:\/\/conversion.com\/blog\/author\/mikestlaurent\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The ROI of Experimentation and Conversion Rate Optimization (CRO) | Conversion","description":"Becoming an experimentation-driven company will improve your ability to innovate, grow, measure, learn, and become customer-centric.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/","og_locale":"en_GB","og_type":"article","og_title":"The ROI of Experimentation and Conversion Rate Optimization (CRO) | Conversion","og_description":"Becoming an experimentation-driven company will improve your ability to innovate, grow, measure, learn, and become customer-centric.","og_url":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/","og_site_name":"Conversion","article_published_time":"2021-04-11T12:00:54+00:00","article_modified_time":"2024-08-20T11:34:13+00:00","og_image":[{"width":1600,"height":1000,"url":"https:\/\/conversion.com\/wp-content\/uploads\/2023\/04\/ROI-of-experimentation_blog_1600x1000_COMPRESSED-1.png","type":"image\/png"}],"author":"Mike St Laurent","twitter_card":"summary_large_image","twitter_image":"https:\/\/conversion.com\/wp-content\/uploads\/2023\/04\/ROI-of-experimentation_blog_1600x1000_COMPRESSED-1.png","twitter_misc":{"Written by":"Mike St Laurent"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/#article","isPartOf":{"@id":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/"},"author":{"name":"Mike St Laurent","@id":"https:\/\/conversion.com\/#\/schema\/person\/af268f8f2dbd6a8eadd8860e089d952d"},"headline":"The ROI of Experimentation and Conversion Rate Optimization (CRO)","datePublished":"2021-04-11T12:00:54+00:00","dateModified":"2024-08-20T11:34:13+00:00","mainEntityOfPage":{"@id":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/"},"wordCount":9,"publisher":{"@id":"https:\/\/conversion.com\/#organization"},"articleSection":["Conversion Optimization","Strategy"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/","url":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/","name":"The ROI of Experimentation and Conversion Rate Optimization (CRO) | Conversion","isPartOf":{"@id":"https:\/\/conversion.com\/#website"},"datePublished":"2021-04-11T12:00:54+00:00","dateModified":"2024-08-20T11:34:13+00:00","description":"Becoming an experimentation-driven company will improve your ability to innovate, grow, measure, learn, and become customer-centric.","breadcrumb":{"@id":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/conversion.com\/blog\/the-roi-of-experimentation-and-conversion-rate-optimization-cro\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/conversion.com\/"},{"@type":"ListItem","position":2,"name":"The ROI of Experimentation and&hellip;"}]},{"@type":"WebSite","@id":"https:\/\/conversion.com\/#website","url":"https:\/\/conversion.com\/","name":"Conversion","description":"Better results with evidence","publisher":{"@id":"https:\/\/conversion.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/conversion.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/conversion.com\/#organization","name":"Conversion","url":"https:\/\/conversion.com\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/","url":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","contentUrl":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","width":1821,"height":193,"caption":"Conversion"},"image":{"@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/conversion.com\/#\/schema\/person\/af268f8f2dbd6a8eadd8860e089d952d","name":"Mike St Laurent","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/4b41d95a6d6a428e6af659935dd55a9e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4b41d95a6d6a428e6af659935dd55a9e?s=96&d=mm&r=g","caption":"Mike St Laurent"},"url":"https:\/\/conversion.com\/blog\/author\/mikestlaurent\/"}]}},"_links":{"self":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/4428"}],"collection":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/users\/44"}],"replies":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/comments?post=4428"}],"version-history":[{"count":5,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/4428\/revisions"}],"predecessor-version":[{"id":7348,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/4428\/revisions\/7348"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media\/4429"}],"wp:attachment":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media?parent=4428"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/categories?post=4428"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}