{"id":200,"date":"2015-06-17T15:24:24","date_gmt":"2015-06-17T15:24:24","guid":{"rendered":"https:\/\/conversion.com\/spotting-patterns-difference-making-losing-money-ab-testing\/"},"modified":"2024-08-13T14:46:15","modified_gmt":"2024-08-13T14:46:15","slug":"spotting-patterns-difference-making-losing-money-ab-testing","status":"publish","type":"post","link":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/","title":{"rendered":"Spotting patterns &#8211; the difference between making and losing money in A\/B testing."},"content":{"rendered":"<section class=\"c-post-content\" data-ref=\"case-content\">\n\t<div class=\"post-content\">\n\t\t<div class=\"post-content__container container container--medium\">\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--links\">\n\t\t\t\t<div class=\"sticky-menu\" data-ref=\"case-content-menu\">\n\t\t\t\t\t<h3 class=\"sticky-menu__title\">Contents<\/h3>\n\t\t\t\t\t<ul class=\"sticky-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link  active\" href=\"#introduction\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Introduction<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#thebigwinner\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">The big winner<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#thebigloser\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">The big loser<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#theclickbait\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">The clickbait<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#thequalifyingchange\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">The qualifying change<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#themessyresult\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">The messy result<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#keytakeaways\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Key takeaways<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t<\/div>\n\t\t\t<\/aside>\n\n\t\t\t<div class=\"post-content__content\">\n\t\t\t\t<ul class=\"post-content__sections\">\n\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"introduction\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><h5>Wrongly interpreting the patterns in your A\/B test results can lose you money. It can lead you to make changes to your site that actually harm your conversion rate.<\/h5>\n<p>&nbsp;<\/p>\n<p>Correctly interpreting\u00a0the patterns in your A\/B test results will mean\u00a0you learn more from each test you run.\u00a0It will will give you confidence that you are only implementing changes that will deliver real revenue impact, and it will help you turn any losing tests into future winners.<\/p>\n<p>At Conversion.com we&#8217;ve run and analyzed hundreds of A\/B and multivariate tests. In our experience, the result of a test will generally fall into one of 5 distinct patterns. We\u2019re going to share these five patterns here, and we\u2019ll tell you what each pattern means in terms of what steps you should take next. Learn to spot these patterns, follow our advice on how to interpret them,\u00a0and you\u2019ll be making the right decision, more often &#8211; making your testing efforts\u00a0more successful.<\/p>\n<p>To illustrate each of the patterns, we\u2019ll imagine we have run an A\/B test on an e-commerce site\u2019s product page and are now looking at the results. We\u2019ll be looking at the increase\/decrease in conversion rate that the new version of this page delivered compared to the original page. We\u2019ll be looking at this on a page-by-page basis for the four steps in the checkout process that the visitor goes through in order to complete their purchase\u00a0(Basket, Checkout, Payment and finally Order Confirmation).<\/p>\n<p>To see the pattern in our results in each case, we\u2019ll plot a simple graph of the conversion rate increase\/decrease to each page. We\u2019ll then look at how this increase\/decrease in conversion rate has changed as we move through our site\u2019s checkout funnel.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__index\">\n\t\t\t\t\t\t\t\t<h3 class=\"post-content__index-title\">Contents<\/h3>\n\t\t\t\t\t\t\t\t<ul class=\"post-content__index-links\" data-ref=\"case-content-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#introduction\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tIntroduction\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#thebigwinner\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tThe big winner\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#thebigloser\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tThe big loser\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#theclickbait\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tThe clickbait\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#thequalifyingchange\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tThe qualifying change\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#themessyresult\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tThe messy result\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#keytakeaways\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tKey takeaways\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"thebigwinner\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">1. The big winner<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>This is the type of test result we all love. Your new version of a page converts at x% higher to the next step than the original and this x% increase continues uniformly all the way to Order Confirmation.<\/p>\n<p>The graph of our first result pattern would look like this.<\/p>\n<p><a href=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_13-10.jpg\"><img decoding=\"async\" class=\"  alignnone wp-image-187 size-full\" src=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_13-10.jpg\" alt=\"The big winner\" width=\"805\" height=\"502\" \/><\/a><\/p>\n<p>We see 10% more visitors reaching each step of the funnel.<\/p>\n<h3>Interpretation<\/h3>\n<p>This pattern is telling us that the new version of the test page successfully encourages 10% more visitors to reach the next step and from there onwards they convert equally as well as existing visitors. The overall result would be a 10% increase in sales. It is clearly logical to implement this new version permanently.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"thebigloser\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">2. The big loser<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>The negative version of this pattern, where each step shows a roughly equal decrease in conversion rate, is a sign that the change that was made has had a clear negative impact. All is not lost though, often an unsuccessful test can be more insightful than a straightforward winner as the negative result forces you to re-evaluate your initial hypothesis and understand what went wrong. You may have stumbled upon a key conversion barrier for your audience and addressing this barrier in the next test could lead to the positive result you have been looking for.<\/p>\n<p>Graphically this pattern will look like this.<\/p>\n<p><a href=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_22-10.jpg\"><img decoding=\"async\" class=\"alignnone wp-image-186 size-full\" src=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_22-10.jpg\" alt=\"The big loser\" width=\"805\" height=\"502\" \/><\/a><\/p>\n<p>We see 10% fewer visitors reaching each step of the funnel.<\/p>\n<h3>Interpretation<\/h3>\n<p>As the opposite of the big winner, this pattern is telling us that the new version of the test page causes\u00a010% fewer visitors to reach the next step and from there onwards they convert equally as well as existing visitors. The overall result would be a 10% decrease\u00a0in sales.\u00a0You would not want to implement this new version of the page.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"theclickbait\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">3. The clickbait<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>\u201cWe increased clickthrus by 307%!\u201d You\u2019ve probably seen sensational headlines like this being thrown around by people in the optimization industry. Hopefully, like us, you\u2019ve developed a strong sense of cynicism when you read results like this. The first question I always ask is \u201cBut how much did sales increase by?\u201d. Chances are, if the result being reported fails to mention the impact on final sales then what they actually saw in their test results was this pattern that we\u2019ve affectionately dubbed \u201cThe clickbait\u201d.<\/p>\n<p>Test results that follow this pattern will show a large increase in the conversion rate to the next step but then this improvement quickly fades away in the later steps and finally there is little or no improvement to Order Confirmation.<\/p>\n<p>Graphically this pattern will look like this.<\/p>\n<p><a href=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_32-10.jpg\"><img decoding=\"async\" class=\"alignnone wp-image-185 size-full\" src=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_32-10.jpg\" alt=\"The clickbait\" width=\"805\" height=\"502\" \/><\/a><\/p>\n<h3>Interpretation<\/h3>\n<p>This pattern catches people out as the large improvement to the next step feels as if it should be a positive result. However, often this pattern of results is merely showing that the new version of the page is pushing a large amount of visitors through to the next step who have no real intention of purchasing. This is illustrated by the sudden large drop in the conversion rate improvement at the later steps when all of the unqualified extra traffic abandons the funnel.<\/p>\n<p>As with all tests, whether this result can be deemed a success depends on the specifics of the site you are testing on and what you are looking to achieve. If there are clear improvements to be made on the next step(s) of the funnel that could help to convert the extra traffic from this test, then it could make sense to address those issues first and then re-run this test. However, if these extra visitors are clicking through by mistake or because they are being misled in any way then you may find it difficult to convert them later no matter what changes you make. Instead, you could be alienating potential customers by delivering a poor customer experience. You\u2019ll also be adding a lot of <i>noise<\/i> to the data of any tests you run on the later pages as there are a lot of extra visitors on those pages who are unlikely to ever purchase.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"thequalifyingchange\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">4. The qualifying change<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>The third pattern is almost the reverse of the second in that here we actually see a drop in conversion to the next step but an overall increase in conversion to order confirmation.<\/p>\n<p>Graphically this pattern looks like this.<\/p>\n<p><a href=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_42-10.jpg\"><img decoding=\"async\" class=\"alignnone wp-image-183 size-full\" src=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_42-10.jpg\" alt=\"The qualifying change\" width=\"805\" height=\"502\" \/><\/a><\/p>\n<h3>Interpretation<\/h3>\n<p>Taking this pattern as a positive can seem counter-intuitive because of the initial drop in conversion to the next step. Arguably, this type of result is actually as good if not better than a big winner from pattern 1. Here the new version of the test page is having what\u2019s known as a qualifying effect. Visitors who may have otherwise abandoned at a later step in the funnel are leaving at the first step instead. Those visitors who do continue past the test page on the other hand are more qualified and therefore convert at a much higher rate. This explains the positive result to Order Confirmation.<\/p>\n<p>Implementing a change that causes this type of pattern means visitors remaining in the funnel now have expressed a clearer desire to purchase. If visitors are still abandoning at a later stage in the funnel, the likelihood now is that this is being caused by a specific weakness on one of those pages. Having removed a lot of the noise from our data, in the form of the unqualified visitors, we are left with a much more reliable measure of the effectiveness of the later steps in the funnel. This means identifying weaknesses in the funnel itself will be far easier.<\/p>\n<p>As with pattern 2 there are circumstances where a result like this may not be preferable. If you already have very low traffic in your funnel then reducing that further could make it even more difficult to get statistically significant results when testing on the later pages of the funnel. You may want to look at tests to drive more traffic to the start of your funnel before implementing a change like this.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"themessyresult\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">5. The messy result<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>This final pattern is often the most difficult to extract insight from as it describes results that show very little pattern whatsoever. Here we often see both increases and decreases in conversion rate to the various steps in the funnel.<\/p>\n<p><a href=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_52-10.jpg\"><img decoding=\"async\" class=\"alignnone wp-image-184 size-full\" src=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/graph_52-10.jpg\" alt=\"The messy result\" width=\"805\" height=\"502\" \/><\/a><\/p>\n<h3>Interpretation<\/h3>\n<p>First and foremost, a lack of a discernible pattern in the results of your split-test can be a tell-tale sign of insufficient levels of data. At the early stages of experiments, when data levels are low, it is not uncommon to see results fluctuating up and down. Reading too much into these results at this stage is a common pitfall. Resist the temptation of checking your experiment results too frequently &#8211; if at all &#8211; in the first few days. Even apparently strong patterns that emerge at these early stages can quickly disappear with a larger sample.<\/p>\n<p>If your test has a large volume of data, and you\u2019re still seeing this type of result, then the likelihood is that your new version of the page is delivering a combination of the effects from the clickbait and the qualifying change patterns. Qualifying some traffic but simultaneously pushing more unqualified traffic through the funnel. If your test involved making multiple changes to a page, try testing the changes separately to pinpoint which individual changes are causing the positive impact and which are causing the negative impact.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"keytakeaways\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Key takeaways<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>The key point to take from all of these patterns is the importance of tracking and analyzing the results at every step of your funnel when you A\/B test, rather than just the next step after your test page. It is easy to see how if only the next step was tracked that many tests can have been falsely declared as winners or losers. In short, this is losing you money.<\/p>\n<p>Detailed test tracking will allow you to pinpoint the exact step in your funnel that visitors are abandoning, and how that differs for each variation of the page that you are testing. This can help to answer the more important question of <i>why<\/i> they are abandoning. If the answer to this is not obvious, running some user tests or watching some recorded user sessions of your test variations can help you to develop these insights and come up with a successful follow up test.<\/p>\n<p>There is a lot more to analyzing A\/B tests than just reading off a conversion rate increase to any single step in your funnel. Often, the pattern of the results can reveal greater insights than the individual numbers. Avoid jumping to conclusions based on a single increase or decrease in conversion to the next step and always track right the way through to the end of your funnel when running tests. Next time you go to analyze a test result, see which of these patterns it matches and consider the implications for your site.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t\n\t\t\t\t<\/ul>\n\t\t\t<\/div>\n\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--form\">\n\t\t\t\t<div class=\"sticky-newsletter\" data-target=\"post-newsletter-anchor\" data-device=\"desktop\">\n                \t<div class=\"post-newsletter\" data-target=\"post-newsletter\">\n    <h4 class=\"post-newsletter__title\">Real-world growth experiments. In your inbox. Every week. <\/h4>\n            <div class=\"post-newsletter__form post-newsletter__form--world kam-world\">\n        <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script> <script>   hbspt.forms.create({     region: \"na1\",     portalId: \"9358319\",     formId: \"4e927e03-4f90-466a-8646-7f94947f860c\"   }); <\/script>\n    <\/div>\n            <div class=\"post-newsletter__form post-newsletter__form--uk-us kam-uk-us\">\n            <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script><script>hbspt.forms.create({ region: \"na1\", portalId: \"9358319\", formId: \"03e799e7-ee63-4857-9182-cb3a687dba40\" });<\/script>\n        <\/div>\n        <div class=\"post-newsletter__terms\"><p>Join 5,000+ optimizers who subscribe to our content<\/p>\n<\/div>\n<\/div>\n\t\t\t\t<\/div>\n            <\/aside>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":8,"featured_media":1395,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[21],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Spotting patterns - the difference between making and losing money in A\/B testing. | Conversion<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Spotting patterns - the difference between making and losing money in A\/B testing. | Conversion\" \/>\n<meta property=\"og:url\" content=\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"Conversion\" \/>\n<meta property=\"article:published_time\" content=\"2015-06-17T15:24:24+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-13T14:46:15+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/spotting-patterns1-12.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"805\" \/>\n\t<meta property=\"og:image:height\" content=\"502\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Kyle Hearnshaw\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kyle Hearnshaw\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/\"},\"author\":{\"name\":\"Kyle Hearnshaw\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/ab0329afc5413d6f2d05195828797be1\"},\"headline\":\"Spotting patterns &#8211; the difference between making and losing money in A\/B testing.\",\"datePublished\":\"2015-06-17T15:24:24+00:00\",\"dateModified\":\"2024-08-13T14:46:15+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/\"},\"wordCount\":14,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"articleSection\":[\"Data &amp; Analytics\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/\",\"url\":\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/\",\"name\":\"Spotting patterns - the difference between making and losing money in A\/B testing. | Conversion\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/#website\"},\"datePublished\":\"2015-06-17T15:24:24+00:00\",\"dateModified\":\"2024-08-13T14:46:15+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/conversion.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Spotting patterns &#8211; the &hellip;\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/conversion.com\/#website\",\"url\":\"https:\/\/conversion.com\/\",\"name\":\"Conversion\",\"description\":\"Better results with evidence\",\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/conversion.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/conversion.com\/#organization\",\"name\":\"Conversion\",\"url\":\"https:\/\/conversion.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"contentUrl\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"width\":1821,\"height\":193,\"caption\":\"Conversion\"},\"image\":{\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/ab0329afc5413d6f2d05195828797be1\",\"name\":\"Kyle Hearnshaw\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a0ca5dc54fade91d56844e7db848011e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a0ca5dc54fade91d56844e7db848011e?s=96&d=mm&r=g\",\"caption\":\"Kyle Hearnshaw\"},\"description\":\"Kyle Hearnshaw is the Head of Conversion Strategy at Conversion.com. He has delivered transformational results for some of the biggest brands through an insight-driven, iterative approach to experimentation.\",\"url\":\"https:\/\/conversion.com\/blog\/author\/kylehearnshaw\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Spotting patterns - the difference between making and losing money in A\/B testing. | Conversion","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/","og_locale":"en_GB","og_type":"article","og_title":"Spotting patterns - the difference between making and losing money in A\/B testing. | Conversion","og_url":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/","og_site_name":"Conversion","article_published_time":"2015-06-17T15:24:24+00:00","article_modified_time":"2024-08-13T14:46:15+00:00","og_image":[{"width":805,"height":502,"url":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/06\/spotting-patterns1-12.jpg","type":"image\/jpeg"}],"author":"Kyle Hearnshaw","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kyle Hearnshaw"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/#article","isPartOf":{"@id":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/"},"author":{"name":"Kyle Hearnshaw","@id":"https:\/\/conversion.com\/#\/schema\/person\/ab0329afc5413d6f2d05195828797be1"},"headline":"Spotting patterns &#8211; the difference between making and losing money in A\/B testing.","datePublished":"2015-06-17T15:24:24+00:00","dateModified":"2024-08-13T14:46:15+00:00","mainEntityOfPage":{"@id":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/"},"wordCount":14,"commentCount":0,"publisher":{"@id":"https:\/\/conversion.com\/#organization"},"articleSection":["Data &amp; Analytics"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/","url":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/","name":"Spotting patterns - the difference between making and losing money in A\/B testing. | Conversion","isPartOf":{"@id":"https:\/\/conversion.com\/#website"},"datePublished":"2015-06-17T15:24:24+00:00","dateModified":"2024-08-13T14:46:15+00:00","breadcrumb":{"@id":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/conversion.com\/blog\/spotting-patterns-difference-making-losing-money-ab-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/conversion.com\/"},{"@type":"ListItem","position":2,"name":"Spotting patterns &#8211; the &hellip;"}]},{"@type":"WebSite","@id":"https:\/\/conversion.com\/#website","url":"https:\/\/conversion.com\/","name":"Conversion","description":"Better results with evidence","publisher":{"@id":"https:\/\/conversion.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/conversion.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/conversion.com\/#organization","name":"Conversion","url":"https:\/\/conversion.com\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/","url":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","contentUrl":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","width":1821,"height":193,"caption":"Conversion"},"image":{"@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/conversion.com\/#\/schema\/person\/ab0329afc5413d6f2d05195828797be1","name":"Kyle Hearnshaw","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a0ca5dc54fade91d56844e7db848011e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a0ca5dc54fade91d56844e7db848011e?s=96&d=mm&r=g","caption":"Kyle Hearnshaw"},"description":"Kyle Hearnshaw is the Head of Conversion Strategy at Conversion.com. He has delivered transformational results for some of the biggest brands through an insight-driven, iterative approach to experimentation.","url":"https:\/\/conversion.com\/blog\/author\/kylehearnshaw\/"}]}},"_links":{"self":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/200"}],"collection":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/comments?post=200"}],"version-history":[{"count":52,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/200\/revisions"}],"predecessor-version":[{"id":6610,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/200\/revisions\/6610"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media\/1395"}],"wp:attachment":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media?parent=200"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/categories?post=200"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}