{"id":811,"date":"2022-04-08T09:40:35","date_gmt":"2022-04-08T09:40:35","guid":{"rendered":"https:\/\/conversion.com\/cro-mistakes\/"},"modified":"2024-08-22T09:52:53","modified_gmt":"2024-08-22T09:52:53","slug":"cro-mistakes","status":"publish","type":"post","link":"https:\/\/conversion.com\/blog\/cro-mistakes\/","title":{"rendered":"The 15 most common CRO mistakes (and how to avoid them)"},"content":{"rendered":"<section class=\"c-post-content\" data-ref=\"case-content\">\n\t<div class=\"post-content\">\n\t\t<div class=\"post-content__container container container--medium\">\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--links\">\n\t\t\t\t<div class=\"sticky-menu\" data-ref=\"case-content-menu\">\n\t\t\t\t\t<h3 class=\"sticky-menu__title\">Contents<\/h3>\n\t\t\t\t\t<ul class=\"sticky-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link  active\" href=\"#introduction\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Introduction<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#startingouttoobig\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Starting out too big<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#averagebuildsizeistoobig\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Average build size is too big<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#testsaretoosmall\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Tests are too small<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#chasingwinners\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Chasing winners<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#nohypothesis\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">No hypothesis<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#statisticalmisunderstandings\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Statistical misunderstandings<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#noresearch\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">No research<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#theflickereffect\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">The Flicker effect<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#wrongprimarymetric\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Wrong primary metric<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#nottrackingguardrailandsecondarymetrics\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Not tracking guardrail and secondary metrics<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#overrelianceonbestpracticesunderrelianceontesting\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Over-reliance on best practices, under reliance on testing<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#notdiggingintosegments\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Not digging into segments<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#littleornoqualityassurance\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Little or no quality assurance<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#noiseofexistingcustomers\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Noise of existing customers<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#sitewideredesigns\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Sitewide redesigns<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#finalthoughts\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Final thoughts<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t<\/div>\n\t\t\t<\/aside>\n\n\t\t\t<div class=\"post-content__content\">\n\t\t\t\t<ul class=\"post-content__sections\">\n\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"introduction\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p><!-- wp:paragraph --><\/p>\n<p>Generating meaningful results through CRO and experimentation is tough.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>While almost anyone can achieve the odd one-off conversion rate uplift, producing wins with any consistency is another challenge entirely.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>What&#8217;s more, many CRO practitioners find that even when they do achieve the kinds of results they&#8217;re after, their winning variations fail to perform when served to 100% of their traffic.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Thankfully, in our experience, almost all CRO-related difficulties can be traced back to a number of core mistakes. In this post, we&#8217;re going to run through a list of the most common (and harmful) ones, explaining what they are and how you can avoid them.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>If we&#8217;ve done our job right, then by the end of this blog, you should have everything you need to sidestep all of these pitfalls and start driving real, replicable results through CRO and experimentation.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>So, to begin: this is our list of mistakes (in no particular order!):<\/p>\n<p><!-- \/wp:paragraph --><\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__index\">\n\t\t\t\t\t\t\t\t<h3 class=\"post-content__index-title\">Contents<\/h3>\n\t\t\t\t\t\t\t\t<ul class=\"post-content__index-links\" data-ref=\"case-content-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#introduction\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tIntroduction\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#startingouttoobig\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tStarting out too big\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#averagebuildsizeistoobig\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tAverage build size is too big\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#testsaretoosmall\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tTests are too small\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#chasingwinners\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tChasing winners\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#nohypothesis\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tNo hypothesis\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#statisticalmisunderstandings\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tStatistical misunderstandings\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#noresearch\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tNo research\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#theflickereffect\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tThe Flicker effect\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#wrongprimarymetric\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tWrong primary metric\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#nottrackingguardrailandsecondarymetrics\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tNot tracking guardrail and secondary metrics\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#overrelianceonbestpracticesunderrelianceontesting\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tOver-reliance on best practices, under reliance on testing\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#notdiggingintosegments\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tNot digging into segments\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#littleornoqualityassurance\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tLittle or no quality assurance\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#noiseofexistingcustomers\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tNoise of existing customers\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#sitewideredesigns\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tSitewide redesigns\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#finalthoughts\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tFinal thoughts\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"startingouttoobig\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">1. Starting out too big<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>As a general rule, it\u2019s never good to start an experimentation program with an ambitious, resource-intensive experiment. Here\u2019s why:<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>A few years ago, we began working with a new client \u2013 a property listing website. Some early research indicated that their property pages might benefit from the inclusion of a maps feature, which would allow their users to see where each property was geographically located on a map.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This functionality was quite complicated to build, and required a lot of time spent on development and QA to ensure that everything was functional and user-friendly.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" class=\"wp-image-3065\" src=\"https:\/\/i1.wp.com\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2022-04-07-at-18.38.56.png?fit=640%2C316&amp;ssl=1\" alt=\"\" \/><\/p>\n<p><div id=\"attachment_5988\" style=\"width: 542px\" class=\"wp-caption aligncenter\"><img aria-describedby=\"caption-attachment-5988\" decoding=\"async\" class=\"wp-image-5988 \" src=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm.png\" alt=\"\" width=\"532\" height=\"272\" srcset=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm.png 2716w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-300x154.png 300w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-1024x524.png 1024w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-768x393.png 768w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-1536x786.png 1536w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-2048x1048.png 2048w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-1600x819.png 1600w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-800x409.png 800w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-400x205.png 400w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-1407x720.png 1407w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-703x360.png 703w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-360x184.png 360w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-920x471.png 920w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-460x235.png 460w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-230x118.png 230w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-115x59.png 115w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-1140x583.png 1140w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-570x292.png 570w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-285x146.png 285w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-142x73.png 142w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-100x51.png 100w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-50x26.png 50w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-25x13.png 25w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-12x6.png 12w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-1670x855.png 1670w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-835x427.png 835w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-417x213.png 417w, https:\/\/conversion.com\/wp-content\/uploads\/2022\/04\/Screenshot-2024-04-30-at-3.21.20\u202fpm-208x106.png 208w\" sizes=\"(max-width: 532px) 100vw, 532px\" \/><\/p>\n<p id=\"caption-attachment-5988\" class=\"wp-caption-text\">An MVE approach would have saved us a lot of time and energy<\/p>\n<\/div>\n<\/figure>\n<p><!-- \/wp:image --> <!-- wp:paragraph --><\/p>\n<p>Unfortunately, when we finally launched our experiment, we found that the variation actually produced a negative impact on our primary conversion metric. In fact, many users were actively navigating away from the maps view that we&#8217;d created!<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This is just the way it goes sometimes. You can never be sure of the impact a change will bring until you\u2019ve tested it. But the mistake we made was sinking too much time and energy into testing a hypothesis that could have been tested using a much simpler, much less resource-expensive experiment.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>For example, we could have used a <a href=\"https:\/\/crstanier.medium.com\/a-product-managers-guide-to-painted-door-tests-a1a5de33b473\">painted door<\/a> test to gauge demand for this new maps feature. This would have been a much quicker experiment to build, and it would have given us everything we needed to validate \u2013 or invalidate \u2013 our hypothesis.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>If we then found that lots of users were trying to use this feature, we could have gone on to build out the functionality with a reasonable degree of certainty that it would actually improve engagement on the site.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>As a result of this mistake, we\u2019ve since adopted the concept of a <strong>minimum viable experiment (MVE) <\/strong>to help guide our experimentation. In essence, an MVE is the smallest possible experiment \u2013 in terms of build, design time, etc. \u2013 that will allow us to validate our hypothesis.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>We now use MVEs to validate our hypotheses at minimal cost and risk, and to gather information about our users which we can use to guide the future course of our programs.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>We\u2019re not saying \u2018don\u2019t spend time on big builds.\u2019 We\u2019re saying \u2018only spend time on big builds when you have a reasonable degree of certainty that those builds are going to be worth it!\u2019<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"averagebuildsizeistoobig\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">2. Average build size is too big<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>This may sound like a similar point to the one raised above, but it\u2019s slightly different. The last point was about not creating experiments with big builds <strong>at the start <\/strong>of your program, when you don\u2019t have much data to support them. This is about creating experiments with big builds in general.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Many people within the CRO industry subscribe to the view that \u2018the bigger the build, the bigger the uplift.\u2019<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>The idea behind this is fairly simple: if I make big changes to a web page, the conversion uplift is likely to be bigger than if I only make small changes.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Anecdotally, though, we\u2019ve always felt that our smaller experiments yielded results that were just as strong as our larger ones \u2013 so we decided to dig into our database, made up of thousands of experiment results, to see what it could tell us.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>The chart below shows our findings.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>As you can see, tweaks have just as high a win rate as experiments with a large build-size \u2013 and they have a slightly greater average uplift too (6.6% vs. 6.5%).<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:image --><\/p>\n<figure class=\"wp-block-image\"><img decoding=\"async\" class=\"aligncenter\" title=\"Chart\" src=\"https:\/\/lh5.googleusercontent.com\/Zt5RFsQgoOu7lmFrUHrtc_qJGuUMv20x6PsxKVo2WMzNXEWeWUW-ZSchOBy8JAZN1SaCQadX7wr6xhPkUl5UnrSfYAvJLGGFx0rWimKE0z1xauTCs9JHJ8mrvxYgkVrvisv0BNSA\" alt=\"\" width=\"560\" height=\"347\" \/><\/figure>\n<p><!-- \/wp:image --> <!-- wp:paragraph --><\/p>\n<p>This data shows that there\u2019s no correlation between build size and either win rate or uplift \u2013 so if you\u2019re spending all of your time building huge experiments that you hope to generate an equally huge uplift, you\u2019re probably going to waste a lot of time.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"testsaretoosmall\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">3. Tests are too small<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>While it\u2019s important to not spend all of your time focusing on huge experiments,\u00a0 it\u2019s also important that you don\u2019t spend all of your time testing minor tweaks either.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>As discussed in the next section on chasing winners, experimentation gives you a chance to trial some of your boldest and brightest ideas \u2013 ideas that have the potential to completely revolutionise the way your business works.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>If all of your experiments are focussed on minor tweaks, you\u2019re missing out on one of the biggest opportunities that CRO offers: taking risks with a safety net.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Ideally, your program will be a combination of small, low-risk tests with a high probability of winning, and higher-risk tests that have the potential to fail horrendously or succeed spectacularly.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"chasingwinners\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">4. Chasing winners<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Following on from our last point: conversion uplifts are important, for sure, but when done right, CRO should also be about gathering deep insights about your customers and trialling bold, innovative ideas with only a fraction of the usual risk.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>As an agency, therefore, if we\u2019re winning too many experiments, we start to ask ourselves if we\u2019re being bold enough.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>A high win rate may look good on paper, but we see it as an indication that we\u2019re being too safe, putting ideas into action that we already have good reason to think will work.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>The most value from CRO comes when you learn things that you didn\u2019t already know \u2013 this allows you to start achieving bigger, more surprising wins, which you can later use to inform not only your experiment roadmap but also your product, pricing, and business strategies too.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"nohypothesis\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">5. No hypothesis<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Many people doing CRO today simply run their tests, analyze the results, look at whether the challenger variation won or lost, and then move on to the next experiment.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>On the one hand, these people should be commended for the fact that they\u2019re running experiments and basing their decisions on empirical evidence. But on the other, their process is missing one of the most important elements of any sound scientific methodology: a hypothesis.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Put simply, every test should be designed to test a hypothesis.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This way, even if your test loses, you\u2019re at least learning something, i.e. that your hypothesis was wrong. You can then use this learning to inform future experiments with an improved chance of success.<\/p>\n<p><!-- \/wp:paragraph --><\/p>\n<p><!-- wp:paragraph --><\/p>\n<p>As will hopefully be obvious by now, high-level CRO is as much about learning as it is about improving your conversion rate. Creating data-backed hypotheses and then testing them is the key to achieving long-term success.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"statisticalmisunderstandings\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">6. Statistical misunderstandings<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Confusion surrounding ab testing statistics is a cause of much CRO-related difficulty.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>For example, many people call (i.e. finish) their tests as soon as they\u2019ve reached 90 or 95% significance.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p><a href=\"http:\/\/www.einarsen.no\/is-your-ab-testing-effort-just-chasing-statistical-ghosts\/\">Mats Einarsen<\/a> showed why this is a bad idea.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>He simulated 1000 A\/A tests (where the control and the variation are identical) and found that 531 one of them reached 95% statistical significance at least once!<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>What this shows is that if you stop your experiment as soon as it reaches a certain significance level \u2013 even if this level is set at 95 or 99% \u2013 there\u2019s a reasonable chance that your result will be the product of blind luck.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>To avoid this mistake, you need to determine your required sample size before you\u2019ve even launched your test \u2013 and you need to stick to it.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Here\u2019s a <a href=\"https:\/\/www.evanmiller.org\/ab-testing\/sample-size.html\">good calculator<\/a> you can use to calculate the sample size you need for your experiment.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>And here\u2019s a <a href=\"https:\/\/cxl.com\/blog\/ab-testing-statistics\/\">good starting point<\/a> to help you learn a bit more about ab testing statistics in general.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"noresearch\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">7. No research<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Having been convinced of the value of CRO and experimentation, the next challenge is deciding what to test.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Should you change your hero image? Should you make your headline copy more emotive? How do you decide which of these ideas to test? What\u2019s more, how do you determine if either of them is worth testing?<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>To answer these questions, you need to do your research.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Research comes in many forms \u2013 analytics audits, scrollmaps, heatmaps, surveys, user testing, biometric testing, etc. \u2013 and it provides you with a good indication as to where and why your web visitors aren\u2019t converting. With this information, you should then have a good idea of which kinds of hypotheses are worth testing and which ought to be pushed further down your list of priorities.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Ultimately, there are any number of potential hypotheses you might want to test on your website. By prioritizing those that are backed by multiple data points, with a mix of qualitative and quantitative research, you\u2019ll be able to zone in on areas of testing that are likely to yield the biggest return.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p><a href=\"https:\/\/conversion.com\/blog\/score-dynamic-prioritization-framework-ab-tests-conversion-com\/\">Learn more about our framework for using research to prioritize a\/b tests.\u00a0<\/a><\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"theflickereffect\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">8. The Flicker effect<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Sometimes when you\u2019re running an a\/b test, the original version of your webpage will appear in the browser before your variant finally \u2018flickers\u2019 into place. This phenomenon is known as the flicker effect (or the flash of original content (FOOC)) and it can play havoc with your experiments.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Not only does it ruin your website\u2019s user experience, but by showing your users both versions of your webpage \u2013 the control and the variation \u2013 it impacts the way they respond to your experiment, invalidating your results.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Thankfully, there are things you can do to <a href=\"https:\/\/www.kameleoon.com\/en\/blog\/ab-testing-flicker-effect\">minimise or entirely remove the flicker effect<\/a>.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Generally speaking, our developers write the code for our clients\u2019 experiments with CRO specific standards in mind. This ensures that the code is executed as quickly as possible, accounting for any time issues, and that the user only sees the version of the webpage that they\u2019re supposed to.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"wrongprimarymetric\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">9. Wrong primary metric<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Your primary metric is the metric that you use to decide whether or not your experiment is a winner or a loser.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>If you have a website that sells shoes, you might set the <em>number of orders<\/em> as your primary metric. This way, if you run an experiment and it results in a 10% uplift in the number of orders, you\u2019ll class it as a winner.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>But what do you do if you\u2019re optimizing a web page that\u2019s a few steps away from your final conversion? For example, maybe you have a four step funnel and you want to optimize the first of these four web pages.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:image --><\/p>\n<figure class=\"wp-block-image\">\n<p><div style=\"width: 549px\" class=\"wp-caption aligncenter\"><img decoding=\"async\" class=\"\" src=\"https:\/\/lh5.googleusercontent.com\/l9yM8j6txcRLVmaEusv986-Xnju4TDHUNneuoo49RGZXciNdHGQzK2w7QhKx0ooGGmUQagfbJ0YdlLhf5-4MXgpfzwlQWpNsXdjE2cLkxCrkwyR6K2DGzCqI89x7-IJAsvWCmTdN\" alt=\"4 step checkout process - made up of a landing page, a basket page, a checkout page, and a confirmation page\" width=\"539\" height=\"214\" \/><\/p>\n<p class=\"wp-caption-text\">4 step checkout process &#8211; made up of a landing page, a basket page, a checkout page, and a confirmation page<\/p>\n<\/div>\n<\/figure>\n<p><!-- \/wp:image --> <!-- wp:paragraph --><\/p>\n<p>What should you use as your primary metric?<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Some people will argue that in this case, you should select the next action you want your user to take as your primary metric, rather than the final conversion. So, in this example, every time a user proceeds from the landing page to the basket page, you would then count it as a conversion.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>But in our experience, this choice of primary metric is a mistake.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>That\u2019s because sometimes, for a variety of reasons, you\u2019ll find that your \u2018next action\u2019 conversion rate goes up but your \u2018final action\u2019 conversion rate goes down.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Take this real-world example:<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>We thought that by making the minibasket easier to use, we would increase the progression rate through to checkout, and that this would ultimately have a positive impact on our final conversion rate.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:image --><\/p>\n<figure class=\"wp-block-image\"><img decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/lh4.googleusercontent.com\/Z9lgPYRfG2pIkvynEcgI0_zt3Vj6kmzN3Bj2DOVno8mq76KGYbU8Ddr3YT2KsyUh19ICY38T9nTQRp8Q8LGbY96IWSdDX2fao6mFtrNvEDdcUeGLUkCAR3o4gzT6cxu1tM4T3TEu\" alt=\"\" width=\"568\" height=\"457\" \/><\/figure>\n<p><!-- \/wp:image --> <!-- wp:paragraph --><\/p>\n<p>However, despite the fact that this experiment increased the progression to checkout rate by 28%, it also increased the dropoff rate on the checkout page by 43%!<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This netted out at a 7.7% decrease in final conversions<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>As a consequence of this result and many others like it, we always recommend using your final conversion as your primary metric.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Note: There are a few occasions when it might actually make sense to set your primary metric as something other than your final conversion. If you\u2019d like to learn about these exceptions to the rule, we discuss this in more detail in our <a href=\"https:\/\/conversion.com\/blog\/the-big-debate-what-should-your-primary-metric-be\/\">blog post about primary metrics.<\/a><\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"nottrackingguardrailandsecondarymetrics\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">10. Not tracking guardrail and secondary metrics<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Selecting the right primary metric is an important first step, but if you want to get as much out of your ab tests as possible, you should also be tracking certain guardrail and secondary metrics too.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This is something that many CRO practitioners fail to do, and it means that they\u2019re leaving all kinds of important insights \u2013 insights that could be used to inform their future testing strategy \u2013 on the table.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p><strong>Guardrail metrics<\/strong> are second tier metrics linked to key business objectives. They help you ensure that your experiment isn&#8217;t inadvertently harming other important business KPI\u2019s.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Here\u2019s an example of the importance of guardrail metrics taken from work we did for one of our clients, a camera vendor:<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>We introduced an \u2018add to basket\u2019 call-to-action (CTA) to the product listing page, which allowed users to make a purchase without having to navigate to the product page.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This test produced a positive uplift on our primary metric \u2013 no. of orders \u2013 but it had a negative impact on two of our guardrail metrics \u2013 average order value (AOV) and revenue.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>If we hadn\u2019t been tracking these guardrail metrics, we would have simply declared this test a winner and recommended that our client served this variation to 100% of their traffic \u2013 costing them a fortune in the process.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Fortunately, on top of tracking guardrail metrics, we were also tracking a number of secondary metrics too. <strong>Secondary metrics<\/strong> don\u2019t determine whether your tests win or lose, but they do allow you to monitor things like engagement, scroll depth, secondary KPI\u2019s, etc., to help you make sense of your result.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>When we dug into our secondary metrics, we found that far fewer users in the variation were purchasing accessories and add-on items than in the control. This was because these users were being diverted away from the product page, which was where they were usually first exposed to these products.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Insights gleaned from these guardrail and secondary metrics not only allowed us to avoid rolling out a new version of the web page that would have harmed business objectives, but they also helped inform the future direction of our testing strategy.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p><a href=\"https:\/\/conversion.com\/blog\/how-to-measure-a-b-tests-for-maximum-impact-and-insight\/\">Learn more about how we measure a\/b tests for maximum impact and insight.\u00a0<\/a><\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"overrelianceonbestpracticesunderrelianceontesting\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">11. Over-reliance on best practices, under reliance on testing<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p><a href=\"https:\/\/conversion.com\/service\/conversion-rate-optimization-services\/\">Conversion rate optimization<\/a>, when done properly, is all about testing hypotheses and making decisions based on the best available evidence. Unfortunately, many people doing CRO today simply apply certain \u2018best practices\u2019 (e.g. CTAs should be in red) to their website, without ever testing whether those best practices are right for them.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>As any good CRO practitioner will know, every website is different.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Just because something works well for some websites doesn\u2019t mean that it will work equally well \u2013 or at all \u2013 for others.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Solely relying on best practices is a recipe for disappointment.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>If you\u2019re serious about optimizing your website, you need to be testing your hypotheses.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"notdiggingintosegments\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">12. Not digging into segments<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Sometimes, for a whole variety of reasons, one variation will perform well with one specific segment but not with another.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>In fact, this is something that we as an agency see all the time.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>A test variation will achieve, say, a 2% conversion uplift, but when we dig into the data, we find that the uplift on mobile was actually +12% while the uplift on desktop was -10%.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This kind of finding has real, quantifiable implications.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>For example, what was it about the variation that mobile users responded so positively to? And why did desktop users respond so poorly? Can we build new experiments to iterate on these findings? Should we serve this new variation page to mobile users but leave the control in place for those using a desktop? Will doing so negatively impact the consistency of the user experience?<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>These are important questions that need to be answered, but unless you\u2019re analyzing your results data and looking at your different segments, you\u2019ll miss them entirely.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"littleornoqualityassurance\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">13. Little or no quality assurance<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>No matter how strong your research or how well designed your experiment, if your webpages aren\u2019t appearing as they should be \u2013 or if they\u2019re appearing differently on different devices, browsers, page resolutions, etc. \u2013 then your results are likely to be skewed.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>One way around this is to invest in rigorous quality assurance (QA).<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>As an agency, we involve our QA engineers right from the start of our process, which allows them to familiarise themselves with each experiment long before it is finally launched. We also encourage them to question everything and to assume, by default, that there will be problems .<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This is a fairly stringent process, but it ensures that our QA engineers are almost always able to catch bugs and usability issues long before the experiment goes live.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>If you\u2019re planning to start running QA of your own, <a href=\"https:\/\/www.browserstack.com\/\">Browserstack<\/a> is a good place to start.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"noiseofexistingcustomers\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">14. Noise of existing customers<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Sometimes you\u2019ll conduct a tonne of research and build a variation page that you\u2019re confident will win, only to find that when you come to test it, the conversion rate has hardly moved and the result hasn\u2019t reached statistical significance.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>What\u2019s happened?<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Well, sometimes, this is just the way it goes. No matter how much research you do, there\u2019s no guarantee that the changes you make will produce their intended effect. This is why testing is so important in the first place.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>But having said this, sometimes there\u2019s another explanation:<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Let\u2019s say that 90% of the people on your website are existing customers. They\u2019ve already been convinced of your product\u2019s value, so when they visit your website, they\u2019re simply there to reorder.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>For these people, the changes you make are unlikely to have much of an impact on their behavior \u2013 they\u2019re going to buy the product regardless of whether you add new imagery, change the headline, etc.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>In situations like this, many people make the mistake of running their tests on existing customers and new users together. When existing customers make up a sizable portion of overall traffic, this has a tendency to muddy the waters and make it extremely difficult for you to achieve a definitive result.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Instead, you often need to find a way to isolate new users to ensure that they\u2019re the only ones being included in your test.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>One way of doing this is to only include new users in your sample (not always possible). Another is to select a primary metric that\u2019s tied exclusively to new user activity \u2013 for example, account creations.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"sitewideredesigns\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">15. Sitewide redesigns<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>This one isn\u2019t necessarily a mistake CRO practitioners themselves make, but it\u2019s something that nonetheless damages conversion rates, so we thought we\u2019d include it here anyway.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Over the years, many companies have come to us with the same problem: feeling that their website\u2019s design had become dated, they\u2019d decided to hire a team of designers and developers to build them something new. They\u2019d spent hundreds of thousands of pounds ensuring that their new website was as sleek and aesthetically pleasing as possible, but when they finally launched it, they found that their conversion rate fell off a cliff.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Many of these redesigns looked great, but for one reason or another, they weren\u2019t performing.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>Ultimately, what\u2019s the point of a fancy website redesign if it harms your bottom line?<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>When we hear that a company is planning to redesign their website, we like to offer an alternative approach: iterative redesign (also sometimes known as evolutionary redesign).<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This approach involves making changes gradually, often one at a time, and running constant tests to see how they\u2019re affecting the website\u2019s conversion rates. If the changes perform well, we keep them and look to build on them; if they perform badly, we reject them and look to learn from their failure.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>This is the method <a href=\"https:\/\/www.versionmuseum.com\/history-of\/amazon-website\">amazon uses,<\/a> and it has allowed them to continually improve their website based on the best available evidence, with little or no risk.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"finalthoughts\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Final thoughts<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Almost anyone can achieve a one-off conversion rate uplift on their website, but generating long-term results through CRO is extremely tough. We hope that this post will give you a good foundational understanding of where you might be going wrong with your CRO efforts and what you can do to start (gracefully!) sidestepping these pitfalls.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>If you\u2019re interested in learning more about how you can use CRO to achieve your business goals, we have a biweekly newsletter where we go into more detail on the various strategies, frameworks, philosophies, and approaches that we\u2019ve used to generate more than \u00a31 billion in additional revenue for our clients.<\/p>\n<p><!-- \/wp:paragraph --> <!-- wp:paragraph --><\/p>\n<p>&nbsp;<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t\n\t\t\t\t<\/ul>\n\t\t\t<\/div>\n\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--form\">\n\t\t\t\t<div class=\"sticky-newsletter\" data-target=\"post-newsletter-anchor\" data-device=\"desktop\">\n                \t<div class=\"post-newsletter\" data-target=\"post-newsletter\">\n    <h4 class=\"post-newsletter__title\">Real-world growth experiments. In your inbox. Every week. <\/h4>\n            <div class=\"post-newsletter__form post-newsletter__form--world kam-world\">\n        <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script> <script>   hbspt.forms.create({     region: \"na1\",     portalId: \"9358319\",     formId: \"4e927e03-4f90-466a-8646-7f94947f860c\"   }); <\/script>\n    <\/div>\n            <div class=\"post-newsletter__form post-newsletter__form--uk-us kam-uk-us\">\n            <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script><script>hbspt.forms.create({ region: \"na1\", portalId: \"9358319\", formId: \"03e799e7-ee63-4857-9182-cb3a687dba40\" });<\/script>\n        <\/div>\n        <div class=\"post-newsletter__terms\"><p>Join 5,000+ optimizers who subscribe to our content<\/p>\n<\/div>\n<\/div>\n\t\t\t\t<\/div>\n            <\/aside>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":40,"featured_media":813,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[29],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>The 15 most common CRO mistakes (and how to avoid them) | Conversion<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/conversion.com\/blog\/cro-mistakes\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The 15 most common CRO mistakes (and how to avoid them) | Conversion\" \/>\n<meta property=\"og:url\" content=\"https:\/\/conversion.com\/blog\/cro-mistakes\/\" \/>\n<meta property=\"og:site_name\" content=\"Conversion\" \/>\n<meta property=\"article:published_time\" content=\"2022-04-08T09:40:35+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-22T09:52:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/Website-blog-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"805\" \/>\n\t<meta property=\"og:image:height\" content=\"503\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Frazer Mawson\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Frazer Mawson\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/conversion.com\/blog\/cro-mistakes\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/blog\/cro-mistakes\/\"},\"author\":{\"name\":\"Frazer Mawson\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/24d093247cc416d047fe90c9f39cfc2a\"},\"headline\":\"The 15 most common CRO mistakes (and how to avoid them)\",\"datePublished\":\"2022-04-08T09:40:35+00:00\",\"dateModified\":\"2024-08-22T09:52:53+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/conversion.com\/blog\/cro-mistakes\/\"},\"wordCount\":10,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"keywords\":[\"ab testing\"],\"articleSection\":[\"General\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/conversion.com\/blog\/cro-mistakes\/\",\"url\":\"https:\/\/conversion.com\/blog\/cro-mistakes\/\",\"name\":\"The 15 most common CRO mistakes (and how to avoid them) | Conversion\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/#website\"},\"datePublished\":\"2022-04-08T09:40:35+00:00\",\"dateModified\":\"2024-08-22T09:52:53+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/conversion.com\/blog\/cro-mistakes\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/conversion.com\/blog\/cro-mistakes\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/conversion.com\/blog\/cro-mistakes\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/conversion.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The 15 most common CRO mistake&hellip;\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/conversion.com\/#website\",\"url\":\"https:\/\/conversion.com\/\",\"name\":\"Conversion\",\"description\":\"Better results with evidence\",\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/conversion.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/conversion.com\/#organization\",\"name\":\"Conversion\",\"url\":\"https:\/\/conversion.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"contentUrl\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"width\":1821,\"height\":193,\"caption\":\"Conversion\"},\"image\":{\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/24d093247cc416d047fe90c9f39cfc2a\",\"name\":\"Frazer Mawson\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ffbec8d1536e460f2be5ffaa0d43a413?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ffbec8d1536e460f2be5ffaa0d43a413?s=96&d=mm&r=g\",\"caption\":\"Frazer Mawson\"},\"url\":\"https:\/\/conversion.com\/blog\/author\/frazermawson\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The 15 most common CRO mistakes (and how to avoid them) | Conversion","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/conversion.com\/blog\/cro-mistakes\/","og_locale":"en_GB","og_type":"article","og_title":"The 15 most common CRO mistakes (and how to avoid them) | Conversion","og_url":"https:\/\/conversion.com\/blog\/cro-mistakes\/","og_site_name":"Conversion","article_published_time":"2022-04-08T09:40:35+00:00","article_modified_time":"2024-08-22T09:52:53+00:00","og_image":[{"width":805,"height":503,"url":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/Website-blog-1.png","type":"image\/png"}],"author":"Frazer Mawson","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Frazer Mawson"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/conversion.com\/blog\/cro-mistakes\/#article","isPartOf":{"@id":"https:\/\/conversion.com\/blog\/cro-mistakes\/"},"author":{"name":"Frazer Mawson","@id":"https:\/\/conversion.com\/#\/schema\/person\/24d093247cc416d047fe90c9f39cfc2a"},"headline":"The 15 most common CRO mistakes (and how to avoid them)","datePublished":"2022-04-08T09:40:35+00:00","dateModified":"2024-08-22T09:52:53+00:00","mainEntityOfPage":{"@id":"https:\/\/conversion.com\/blog\/cro-mistakes\/"},"wordCount":10,"commentCount":0,"publisher":{"@id":"https:\/\/conversion.com\/#organization"},"keywords":["ab testing"],"articleSection":["General"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/conversion.com\/blog\/cro-mistakes\/","url":"https:\/\/conversion.com\/blog\/cro-mistakes\/","name":"The 15 most common CRO mistakes (and how to avoid them) | Conversion","isPartOf":{"@id":"https:\/\/conversion.com\/#website"},"datePublished":"2022-04-08T09:40:35+00:00","dateModified":"2024-08-22T09:52:53+00:00","breadcrumb":{"@id":"https:\/\/conversion.com\/blog\/cro-mistakes\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/conversion.com\/blog\/cro-mistakes\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/conversion.com\/blog\/cro-mistakes\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/conversion.com\/"},{"@type":"ListItem","position":2,"name":"The 15 most common CRO mistake&hellip;"}]},{"@type":"WebSite","@id":"https:\/\/conversion.com\/#website","url":"https:\/\/conversion.com\/","name":"Conversion","description":"Better results with evidence","publisher":{"@id":"https:\/\/conversion.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/conversion.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/conversion.com\/#organization","name":"Conversion","url":"https:\/\/conversion.com\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/","url":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","contentUrl":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","width":1821,"height":193,"caption":"Conversion"},"image":{"@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/conversion.com\/#\/schema\/person\/24d093247cc416d047fe90c9f39cfc2a","name":"Frazer Mawson","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/ffbec8d1536e460f2be5ffaa0d43a413?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ffbec8d1536e460f2be5ffaa0d43a413?s=96&d=mm&r=g","caption":"Frazer Mawson"},"url":"https:\/\/conversion.com\/blog\/author\/frazermawson\/"}]}},"_links":{"self":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/811"}],"collection":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/comments?post=811"}],"version-history":[{"count":17,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/811\/revisions"}],"predecessor-version":[{"id":7427,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/811\/revisions\/7427"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media\/813"}],"wp:attachment":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media?parent=811"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/categories?post=811"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}