{"id":4530,"date":"2021-04-11T11:59:22","date_gmt":"2021-04-11T11:59:22","guid":{"rendered":"https:\/\/conversion.com\/?p=4530"},"modified":"2024-08-21T13:19:44","modified_gmt":"2024-08-21T13:19:44","slug":"3-mistakes-invalidate-ab-test-results","status":"publish","type":"post","link":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/","title":{"rendered":"The top 3 mistakes that make your A\/B test results invalid"},"content":{"rendered":"<section class=\"c-post-content\" data-ref=\"case-content\">\n\t<div class=\"post-content\">\n\t\t<div class=\"post-content__container container container--medium\">\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--links\">\n\t\t\t\t<div class=\"sticky-menu\" data-ref=\"case-content-menu\">\n\t\t\t\t\t<h3 class=\"sticky-menu__title\">Contents<\/h3>\n\t\t\t\t\t<ul class=\"sticky-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link  active\" href=\"#introduction\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Introduction<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#yourtesthastoomanyvariations\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Your test has too many variations<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#youchangeexperimentsettingsinthemiddleofatest\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">You change experiment settings in the middle of a test<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#youredoingposttestsegmentationincorrectly\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">You\u2019re doing post-test segmentation incorrectly<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t                            \t\t\t\t\t\t\t<li class=\"sticky-menu__item\">\n\t\t\t\t\t\t\t\t<a class=\"sticky-menu__item-link \" href=\"#conclusion\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t<span class=\"arrow\"><svg viewBox=\"0 0 16 20\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M10.6875 9.34375V7.34375L13.3438 10L10.6875 12.6562V10.6562H2.65625V9.34375H10.6875Z\" fill=\"#F6876F\"\/><\/svg><\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"sticky-menu__item-text\">Conclusion<\/span>\n\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t<\/div>\n\t\t\t<\/aside>\n\n\t\t\t<div class=\"post-content__content\">\n\t\t\t\t<ul class=\"post-content__sections\">\n\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"introduction\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p><strong>A Fortune 500 company asked that I review their A\/B testing strategy.<\/strong><\/p>\n<p>The results were good, the hypotheses strong, everything seemed to be in order\u2026 until I looked at the log of changes in their experimentation tool.<\/p>\n<p>I noticed several blunders: In some experiments, they had adjusted the traffic allocation for the variations mid-experiment; some variations had been paused for a few days, then resumed; and experiments were stopped as soon as statistical significance was reached.<\/p>\n<p>When it comes to testing, too many companies worry about the \u201cwhat\u201d, or the design of their variations, and not enough worry about the \u201chow\u201d,\u00a0<strong>the execution of their experiments.<\/strong><\/p>\n<p>Don\u2019t get me wrong, variation design is important: you need solid hypotheses supported by strong evidence. However, if you believe your work is finished once you have come up with variations for an experiment and pressed the launch button, you\u2019re wrong.<\/p>\n<p>In fact,\u00a0<strong>the way you run your A\/B tests<\/strong>\u00a0is the most difficult and most important piece of the optimization puzzle.<\/p>\n<blockquote>\n<p>\u201dThere are three kinds of lies: lies, damned lies, and statistics.\u201d<\/p>\n<p>\u2013 Mark Twain<\/p>\n<\/blockquote>\n<p>In this post, I will share the biggest mistakes you can make within each step of the testing process: the design, launch, and analysis of an experiment, and how to avoid them.\u00a0<strong>This post is fairly technical. Here\u2019s how you should read it:<\/strong><\/p>\n<ul>\n<li>If you are just getting started with conversion optimization (CRO), or are not directly involved in designing or analyzing tests, feel free to skip the more technical sections and simply skim for insights.<\/li>\n<li>If you are an expert in CRO or are involved in designing and analyzing tests, you will want to pay attention to the technical details. These sections are highlighted in blue.<\/li>\n<\/ul>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__index\">\n\t\t\t\t\t\t\t\t<h3 class=\"post-content__index-title\">Contents<\/h3>\n\t\t\t\t\t\t\t\t<ul class=\"post-content__index-links\" data-ref=\"case-content-menu\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#introduction\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tIntroduction\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#yourtesthastoomanyvariations\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tYour test has too many variations\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#youchangeexperimentsettingsinthemiddleofatest\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tYou change experiment settings in the middle of a test\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#youredoingposttestsegmentationincorrectly\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tYou\u2019re doing post-test segmentation incorrectly\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"post-content__index-link\">\n\t\t\t\t\t\t\t\t\t\t\t<a href=\"#conclusion\" data-ref=\"case-content-menu-item\">\n\t\t\t\t\t\t\t\t\t\t\t\tConclusion\n\t\t\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"yourtesthastoomanyvariations\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Mistake #1: Your test has too many variations<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>The more variations, the more insights you\u2019ll get, right?<\/p>\n<p>Not exactly. Having too many variations slows down your tests but, more importantly, it can impact the integrity of your data in 2 ways.<\/p>\n<p><strong>First<\/strong>, the more variations you test against each other, the more traffic you will need, and the longer you\u2019ll have to run your test to get results that you can trust. This is simple math.<\/p>\n<p>But the issue with running a longer test is that you are more likely to be exposed to cookie deletion. If you run an A\/B test for more than 3\u20134 weeks, the risk of sample pollution increases: in that time, people will have deleted their cookies and may enter a different variation than the one they were originally in.<\/p>\n<blockquote>\n<p>\u201dWithin 2 weeks, you can get a 10% dropout of people deleting cookies and that can really affect your sample quality.\u201d<\/p>\n<p>\u2013 Ton Wesseling, Founder,\u00a0<a href=\"http:\/\/onlinedialogue.com\/\">Online Dialogue<\/a><\/p>\n<\/blockquote>\n<p>The\u00a0<strong>second risk<\/strong>\u00a0when testing multiple variations is that the significance level goes down as the number of variations increases.<\/p>\n<p>For example, if you use the accepted significance level of 0.05 and decide to test 20 different scenarios, one of those will be significant purely by chance (20 * 0.05). If you test 100 different scenarios, the number goes up to five (100 * 0.05).<\/p>\n<p>In other words, the more variations, the higher the chance of a false positive i.e. the higher your chances of finding a winner that is not significant.<\/p>\n<p><a href=\"https:\/\/www.theguardian.com\/technology\/2014\/feb\/05\/why-google-engineers-designers\" target=\"blank\" rel=\"noopener\">Google\u2019s 41 shades of blue<\/a>\u00a0is a good example of this. In 2009, when Google could not decide which shades of blue would generate the most clicks on their search results page, they decided to test 41 shades. At a 95% confidence level, the chance of getting a false positive was 88%. If they had tested 10 shades, the chance of getting a false positive would have been 40%, 9% with 3 shades, and down to 5% with 2 shades.<\/p>\n<p>This is called the\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Multiple_comparisons_problem\" target=\"blank\" rel=\"noopener\">Multiple Comparison Problem<\/a>.<\/p>\n<figure class=\"alignfull\">\n<div class=\" container\">\n<div class=\" row\">\n<div class=\"offset-md-2 col-md-8 col-md-8\">\n<p>You can calculate the chance of getting a false positive using the following formula: 1-(1-<em>a<\/em>)^m with m being the total number of variations tested and\u00a0<em>a<\/em>\u00a0being the significance level. With a significance level of 0.05, the equation would look like this:1-(1-0.05)^m or 1-0.95^m.<\/p>\n<p>You can fix the multiple comparison problem using the Bonferroni correction, which calculates the confidence level for an individual test when more than one variation or hypothesis is being tested.<\/p>\n<p>Wikipedia illustrates the Bonferroni correction with the following example: \u201cIf an experimenter is testing m hypotheses, [and] the desired significance level for the whole family of tests is a, then the Bonferroni correction would test each individual hypothesis at a significance level of a\/m.<\/p>\n<p>For example, if [you are] testing m = 8 hypotheses with a desired a = 0.05, then the Bonferroni correction would test each individual hypothesis at a = 0.05\/8=0.00625.\u201d<br \/>\nIn other words, you\u2019ll need a 0.625% significance level, which is the same as a 99.375% confidence level (100% \u2013 0.625%) for an individual test.<\/p>\n<p>The Bonferroni correction tends to be a bit too conservative and is based on the assumption that all tests are independent of each other. However, it demonstrates how multiple comparisons can skew your data if you don\u2019t adjust the significance level accordingly.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/figure>\n<p>The following tables summarize the multiple comparison problem.<\/p>\n<p><strong>Probability of a false positive with a 0.05 significance level:<\/strong><\/p>\n<div class=\"main-content__image-component\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204840\/False-Positive.png\" alt=\"Probability of a false positive with a 0.05 significance level:\" width=\"409\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204840\/False-Positive.png\" data-ll-status=\"loaded\" \/><\/div>\n<p><strong>Adjusted significance and confidence levels to maintain a 5% false discovery probability:<\/strong><\/p>\n<div class=\"main-content__image-component\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204838\/Confidence-levels-700x219-1-640x200-1.png\" alt=\"Adjusted significance and confidence levels to maintain a 5% false discovery probability:\" width=\"598\" height=\"187\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204838\/Confidence-levels-700x219-1-640x200-1.png\" data-ll-status=\"loaded\" \/><\/div>\n<p>In this section, I\u2019m talking about the risks of testing a high number of variations in an experiment. But the same problem also applies when you test multiple goals and segments, which we\u2019ll review a bit later.<\/p>\n<blockquote>\n<p>\u201dEach additional variation and goal adds a new combination of individual statistics for online experiments comparisons to an experiment. In a scenario where there are four variations and four goals, that\u2019s 16 potential outcomes that need to be controlled for separately.\u201d<\/p>\n<p>\u2013\u00a0<a href=\"http:\/\/pages.optimizely.com\/rs\/optimizely\/images\/practical_guide_to_stats.pdf\">Optimizely Practical Guide to Stats<\/a><\/p>\n<\/blockquote>\n<p>Some A\/B testing tools, such as VWO and Optimizely, adjust for the multiple comparison problem. These tools will make sure that the false positive rate of your experiment matches the false positive rate you think you are getting.<\/p>\n<p>In other words, the false positive rate you set in your significance threshold will reflect the true chance of getting a false positive: you won\u2019t need to correct and adjust the confidence level using the Bonferroni or any other methods.<\/p>\n<p>One final problem with testing multiple variations can occur when you are analyzing the results of your test. You may be tempted to declare the variation with the highest lift the winner, even though there is no statistically significant difference between the winner and the runner up. This means that, even though one variation may be performing better in the current test, the runner up could \u201cwin\u201d in the next round.<\/p>\n<p>You should consider both variations as winners.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"youchangeexperimentsettingsinthemiddleofatest\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Mistake #2: You change experiment settings in the middle of a test<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>When you launch an experiment, you need to commit to it fully. Do not change the experiment settings, the test goals, the design of the variation or of the Control mid-experiment. And don\u2019t change traffic allocations to variations.<\/p>\n<p>Changing the traffic split between variations during an experiment will impact the integrity of your results because of a problem known as\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Simpson%27s_paradox\" target=\"blank\" rel=\"noopener\">Simpson\u2019s Paradox<\/a>.This statistical paradox appears when we see a trend in different groups of data which disappears when those groups are combined.<\/p>\n<figure class=\"alignfull\">\n<div class=\"container\">\n<div class=\"row\">\n<div class=\"offset-md-2 col-md-8 col-md-8\">\n<p>Ronny Kohavi from Microsoft shares an example wherein a website gets one million daily visitors, on both Friday and Saturday. On Friday, 1% of the traffic is assigned to the treatment (i.e. the variation), and on Saturday that percentage is raised to 50%.<\/p>\n<p>Even though the treatment has a higher conversion rate than the Control on both Friday (2.30% vs. 2.02%) and Saturday (1.2% vs. 1.00%), when the data is combined over the two days, the treatment seems to underperform (1.20% vs. 1.68%).<\/p>\n<p>This is because we are dealing with weighted averages. The data from Saturday, a day with an overall worse conversion rate, impacted the treatment more than that from Friday.<\/p>\n<div class=\"main-content__image-component\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204843\/Table-1.png\" alt=\"Conversion rate for two days. Each day has 1M customers, and the Treatment (T) is better than the Control (C) on each day, yet worse overall.\" width=\"583\" height=\"240\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204843\/Table-1.png\" data-ll-status=\"loaded\" \/><\/div>\n<p>Source:\u00a0<a href=\"http:\/\/www.exp-platform.com\/Documents\/2009-ExPpitfalls.pdf\">Seven Pitfalls to Avoid when Running Controlled Experiments on the Web<\/a><\/p>\n<p>We will return to Simpson\u2019s Paradox in just a bit.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/figure>\n<p>Changing the traffic allocation mid-test will also skew your results because it alters the sampling of your returning visitors.<\/p>\n<p>Changes made to the traffic allocation only affect new users. Once visitors are bucketed into a variation, they will continue to see that variation for as long as the experiment is running.<\/p>\n<p>So, let\u2019s say you start a test by allocating 80% of your traffic to the Control and 20% to the variation. Then, after a few days you change it to a 50\/50 split. All new users will be allocated accordingly from then on.<\/p>\n<p>However, all the users that entered the experiment prior to the change will be bucketed into the same variation they entered previously. In our current example, this means that the returning visitors will still be assigned to the Control and you will now have a large proportion of returning visitors (who are more likely to convert) in the Control.<\/p>\n<p><em>Note: This problem of changing traffic allocation mid-test only happens if you make a change at the variation level. You can change the traffic allocation at the experiment level mid-experiment. This is useful if you want to have a ramp up period where you target only 50% of your traffic for the first few days of a test before increasing it to 100%. This won\u2019t impact the integrity of your results.<\/em><\/p>\n<p>As I mentioned earlier, the \u201cdo not change mid-test rule\u201d extends to your test goals and the designs of your variations. If you\u2019re tracking multiple goals during an experiment, you may be tempted to change what the main goal should be mid-experiment. Don\u2019t do it.<\/p>\n<p>All Optimizers have a favorite variation that we secretly hope will win during any given test. This is not a problem until you start giving weight to the metrics that favor this variation. Decide on a goal metric that you can measure in the short term (the duration of a test) and that can predict your success in the long term. Track it and stick to it.<\/p>\n<p>It is useful to track other key metrics to gain insights and\/or debug an experiment, if something looks wrong. However, these are not the metrics you should look at to make a decision, even though they may favor your favorite variation.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"youredoingposttestsegmentationincorrectly\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h2 class=\"post-content__section-title\">Mistake #3: You\u2019re doing post-test segmentation incorrectly<\/h2>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>Let\u2019s say you have avoided the 2 mistakes I\u2019ve already discussed, and you\u2019re pretty confident about the results you see in your A\/B testing tool. It\u2019s time to analyze the results, right?<\/p>\n<p>Not so fast! Did you stop the test as soon as it reached statistical significance?<\/p>\n<p>I hope not\u2026<\/p>\n<p><strong>Statistical significance should not dictate when you stop a test.<\/strong>\u00a0It only tells you if there is a difference between your Control and your variations. This is why you should not wait for a test to be significant (because it may never happen) or stop a test as soon as it is significant. Instead, you need to wait for the calculated sample size to be reached before stopping a test. Use a\u00a0<a href=\"https:\/\/vwo.com\/ab-split-test-duration\/\" target=\"blank\" rel=\"noopener\">test duration calculator<\/a>\u00a0to understand better when to stop a test.<\/p>\n<div class=\"main-content__image-component\">\n<div style=\"width: 609px\" class=\"wp-caption aligncenter\"><img decoding=\"async\" class=\"entered lazyloaded\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/07\/09213716\/Screen-Shot-2021-07-09-at-2.34.31-PM.png\" alt=\"\" width=\"599\" height=\"494\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/07\/09213716\/Screen-Shot-2021-07-09-at-2.34.31-PM.png\" data-ll-status=\"loaded\" \/><\/p>\n<p class=\"wp-caption-text\">A test duration calculator like this one from VWO can help you determine how long to run an A\/B test for.<\/p>\n<\/div>\n<\/div>\n<p>Now, assuming you\u2019ve stopped your test at the correct time, we can move on to segmentation. Segmentation and personalization are hot topics in marketing right now, and more and more tools enable segmentation and personalization.<\/p>\n<p>It follows then, that after you stop a test, you probably start dissecting the results based on segments such as traffic source, new vs. returning users, device type, etc. This technique is called post-test segmentation and is one of the\u00a03 ways to create personalization and segmentation hypotheses to test.<\/p>\n<p>There are 2 main problems with post-test segmentation, however, that will impact the statistical validity of your segments (when done incorrectly).<\/p>\n<ol>\n<li>The sample size of your segments is too small. You stopped the test when you reached the calculated sample size, but at a segment level the sample size is likely too small and the lift between segments has no statistical validity.<\/li>\n<li>The multiple comparison problem. The more segments you compare, the greater the likelihood that you\u2019ll get a false positive among those tests. With a 95% confidence level, you\u2019re likely to get a false positive every 20 post-test segments you look at.<\/li>\n<\/ol>\n<p>There are different ways to prevent these two issues, but the easiest and most accurate strategy is to create targeted tests (rather than breaking down results per segment post-test).<\/p>\n<p>I don\u2019t advocate against post-test segmentation\u2015quite the opposite. In fact, looking at too much aggregate data can be misleading. (Simpson\u2019s Paradox strikes back.)<\/p>\n<p>The Wikipedia definition for Simpson\u2019s Paradox provides a real-life example from a medical study comparing the success rates of two treatments for kidney stones.The table below shows the success rates and numbers of treatments for treatments involving both small and large kidney stones.)<\/p>\n<figure class=\"alignfull\">\n<div class=\"container\">\n<div class=\"row\">\n<div class=\"offset-md-2 col-md-8 col-md-8\">\n<div class=\"main-content__image-component\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204842\/Stones.png\" alt=\"The table shows the success rates and numbers of treatments for treatments involving both small and large kidney stones.\" width=\"438\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204842\/Stones.png\" data-ll-status=\"loaded\" \/><\/div>\n<p>Source:\u00a0<a href=\"http:\/\/www.bmj.com\/content\/309\/6967\/1480.full\" target=\"blank\" rel=\"noopener\">Confounding and Simpson\u2019s Paradox<\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/figure>\n<p>The paradoxical conclusion is that treatment A is more effective when used on small stones, and also when used on large stones, yet treatment B is more effective when considering both sizes at the same time.<\/p>\n<p>In the context of an A\/B test, this would look something like this:<\/p>\n<div class=\"main-content__image\"><img decoding=\"async\" class=\"entered lazyloaded aligncenter\" src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204838\/Aggregate-data-1-700x111-1-640x101-1.png\" alt=\"The table shows data segmented by traffic source.\" width=\"589\" height=\"93\" data-lazy-src=\"https:\/\/cdn.conversion.com\/wp-content\/uploads\/2021\/04\/20204838\/Aggregate-data-1-700x111-1-640x101-1.png\" data-ll-status=\"loaded\" \/><\/div>\n<p>Source:\u00a0<a href=\"http:\/\/blog.analytics-toolkit.com\/2014\/segmenting-data-web-analytics-simpsons-paradox\/\" target=\"blank\" rel=\"noopener\">Segmenting Data for Web Analytics \u2013 The Simpson\u2019s Paradox<\/a><\/p>\n<p>Simpson\u2019s Paradox surfaces when sampling is not uniform\u2014that is the sample size of your segments is different. There are a few things you can do to prevent getting lost in and misled by this paradox.<\/p>\n<p>First, you can prevent this problem from happening altogether by using\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Stratified_sampling\" target=\"blank\" rel=\"noopener\">stratified sampling<\/a>, which is the process of dividing members of the population into homogeneous and mutually exclusive subgroups before sampling. However, most tools don\u2019t offer this option.<\/p>\n<p>If you are already in a situation where you have to decide whether to act on aggregate data or on segment data,\u00a0<a href=\"http:\/\/blog.analytics-toolkit.com\/2014\/segmenting-data-web-analytics-simpsons-paradox\/\" target=\"blank\" rel=\"noopener\">Georgi Georgiev<\/a>\u00a0recommends you look at the story behind the numbers, rather than at the numbers themselves.<\/p>\n<p>\u201cMy recommendation in the specific example [illustrated in the table above] is to refrain from making a decision with the data in the table. Instead, we should consider looking at each traffic source\/landing page couple from a qualitative standpoint first. Based on the nature of each traffic source (one-time, seasonal, stable) we might reach a different final decision. For example, we may consider retaining both landing pages, but for different sources.<\/p>\n<p>In order to do that in a data-driven manner, we should treat each source\/page couple as a separate test variation and perform some additional testing until we reach the desired statistically significant result for each pair (currently we do not have significant results pair-wise).\u201d<\/p>\n<p>In a nutshell, it can be complicated to get post-test segmentation right, but when you do, it will unveil insights that your aggregate data can\u2019t. Remember, you will have to validate the data for each segment in a separate follow up test.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t                        \t\t\t\t\t\t<li class=\"post-content__section\" id=\"conclusion\" data-ref=\"case-content-item\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"post-content__section-content\"><p>The execution of an experiment is the most important part of a successful optimization strategy. If your tests are not executed properly, your results will be invalid and you will be relying on misleading data.<\/p>\n<p>It is always tempting to showcase good results. Results are often the most important factor when your boss is evaluating the success of your conversion optimization department or agency.<\/p>\n<p>But results aren\u2019t always trustworthy. Too often, the numbers you see in case studies are lacking valid statistical inferences: either they rely too heavily on an A\/B testing tool\u2019s unreliable stats engine and\/or they haven\u2019t addressed the common pitfalls outlined in this post.<\/p>\n<p>Use case studies as a source of inspiration, but make sure that you are executing your tests properly by doing the following:<\/p>\n<ul>\n<li>If your A\/B testing tool doesn\u2019t adjust for the multiple comparison problem, make sure to correct your significance level for tests with more than 1 variation<\/li>\n<li>Don\u2019t change your experiment settings mid-experiment<\/li>\n<li>Don\u2019t use statistical significance as an indicator of when to stop a test, and make sure to calculate the sample size you need to reach before calling a test complete<\/li>\n<li>Finally, keep segmenting your data post-test. But make sure you are not falling into the multiple comparison trap and are comparing segments that are significant and have a big enough sample size<\/li>\n<\/ul>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\n\t\t\t\t\t\t                            <div data-target=\"post-newsletter-anchor\" data-device=\"mobile\"><\/div>\n                        \t\t\t\t\t\n\t\t\t\t<\/ul>\n\t\t\t<\/div>\n\n\t\t\t<aside class=\"post-content__sidebar post-content__sidebar--form\">\n\t\t\t\t<div class=\"sticky-newsletter\" data-target=\"post-newsletter-anchor\" data-device=\"desktop\">\n                \t<div class=\"post-newsletter\" data-target=\"post-newsletter\">\n    <h4 class=\"post-newsletter__title\">Real-world growth experiments. In your inbox. Every week. <\/h4>\n            <div class=\"post-newsletter__form post-newsletter__form--world kam-world\">\n        <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script> <script>   hbspt.forms.create({     region: \"na1\",     portalId: \"9358319\",     formId: \"4e927e03-4f90-466a-8646-7f94947f860c\"   }); <\/script>\n    <\/div>\n            <div class=\"post-newsletter__form post-newsletter__form--uk-us kam-uk-us\">\n            <script charset=\"utf-8\" type=\"text\/javascript\" src=\"\/\/js.hsforms.net\/forms\/embed\/v2.js\"><\/script><script>hbspt.forms.create({ region: \"na1\", portalId: \"9358319\", formId: \"03e799e7-ee63-4857-9182-cb3a687dba40\" });<\/script>\n        <\/div>\n        <div class=\"post-newsletter__terms\"><p>Join 5,000+ optimizers who subscribe to our content<\/p>\n<\/div>\n<\/div>\n\t\t\t\t<\/div>\n            <\/aside>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":62,"featured_media":4696,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[12],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>The top 3 mistakes that make your A\/B test results invalid | Conversion<\/title>\n<meta name=\"description\" content=\"Running your A\/B tests is the most difficult and most important piece of the optimization puzzle. Avoid these 3 mistakes to run your A\/B tests properly.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The top 3 mistakes that make your A\/B test results invalid | Conversion\" \/>\n<meta property=\"og:description\" content=\"Running your A\/B tests is the most difficult and most important piece of the optimization puzzle. Avoid these 3 mistakes to run your A\/B tests properly.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/\" \/>\n<meta property=\"og:site_name\" content=\"Conversion\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-11T11:59:22+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-21T13:19:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/conversion.com\/wp-content\/uploads\/2021\/04\/top-3-mistakes_blog_1600x1000_COMPRESSED-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1600\" \/>\n\t<meta property=\"og:image:height\" content=\"1000\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"James Flory\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/conversion.com\/wp-content\/uploads\/2021\/04\/top-3-mistakes_blog_1600x1000_COMPRESSED-1.png\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"James Flory\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/\"},\"author\":{\"name\":\"James Flory\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/843da25bf5d8480f182badf116cc1b4d\"},\"headline\":\"The top 3 mistakes that make your A\/B test results invalid\",\"datePublished\":\"2021-04-11T11:59:22+00:00\",\"dateModified\":\"2024-08-21T13:19:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/\"},\"wordCount\":11,\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"articleSection\":[\"Conversion Optimization\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/\",\"url\":\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/\",\"name\":\"The top 3 mistakes that make your A\/B test results invalid | Conversion\",\"isPartOf\":{\"@id\":\"https:\/\/conversion.com\/#website\"},\"datePublished\":\"2021-04-11T11:59:22+00:00\",\"dateModified\":\"2024-08-21T13:19:44+00:00\",\"description\":\"Running your A\/B tests is the most difficult and most important piece of the optimization puzzle. Avoid these 3 mistakes to run your A\/B tests properly.\",\"breadcrumb\":{\"@id\":\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/conversion.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The top 3 mistakes that make y&hellip;\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/conversion.com\/#website\",\"url\":\"https:\/\/conversion.com\/\",\"name\":\"Conversion\",\"description\":\"Better results with evidence\",\"publisher\":{\"@id\":\"https:\/\/conversion.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/conversion.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/conversion.com\/#organization\",\"name\":\"Conversion\",\"url\":\"https:\/\/conversion.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"contentUrl\":\"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png\",\"width\":1821,\"height\":193,\"caption\":\"Conversion\"},\"image\":{\"@id\":\"https:\/\/conversion.com\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/843da25bf5d8480f182badf116cc1b4d\",\"name\":\"James Flory\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/conversion.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/30287cbe25c2d5bb42ba74b00f2223af?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/30287cbe25c2d5bb42ba74b00f2223af?s=96&d=mm&r=g\",\"caption\":\"James Flory\"},\"url\":\"https:\/\/conversion.com\/blog\/author\/james-flory\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The top 3 mistakes that make your A\/B test results invalid | Conversion","description":"Running your A\/B tests is the most difficult and most important piece of the optimization puzzle. Avoid these 3 mistakes to run your A\/B tests properly.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/","og_locale":"en_GB","og_type":"article","og_title":"The top 3 mistakes that make your A\/B test results invalid | Conversion","og_description":"Running your A\/B tests is the most difficult and most important piece of the optimization puzzle. Avoid these 3 mistakes to run your A\/B tests properly.","og_url":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/","og_site_name":"Conversion","article_published_time":"2021-04-11T11:59:22+00:00","article_modified_time":"2024-08-21T13:19:44+00:00","og_image":[{"width":1600,"height":1000,"url":"https:\/\/conversion.com\/wp-content\/uploads\/2021\/04\/top-3-mistakes_blog_1600x1000_COMPRESSED-1.png","type":"image\/png"}],"author":"James Flory","twitter_card":"summary_large_image","twitter_image":"https:\/\/conversion.com\/wp-content\/uploads\/2021\/04\/top-3-mistakes_blog_1600x1000_COMPRESSED-1.png","twitter_misc":{"Written by":"James Flory"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/#article","isPartOf":{"@id":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/"},"author":{"name":"James Flory","@id":"https:\/\/conversion.com\/#\/schema\/person\/843da25bf5d8480f182badf116cc1b4d"},"headline":"The top 3 mistakes that make your A\/B test results invalid","datePublished":"2021-04-11T11:59:22+00:00","dateModified":"2024-08-21T13:19:44+00:00","mainEntityOfPage":{"@id":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/"},"wordCount":11,"publisher":{"@id":"https:\/\/conversion.com\/#organization"},"articleSection":["Conversion Optimization"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/","url":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/","name":"The top 3 mistakes that make your A\/B test results invalid | Conversion","isPartOf":{"@id":"https:\/\/conversion.com\/#website"},"datePublished":"2021-04-11T11:59:22+00:00","dateModified":"2024-08-21T13:19:44+00:00","description":"Running your A\/B tests is the most difficult and most important piece of the optimization puzzle. Avoid these 3 mistakes to run your A\/B tests properly.","breadcrumb":{"@id":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/conversion.com\/blog\/3-mistakes-invalidate-ab-test-results\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/conversion.com\/"},{"@type":"ListItem","position":2,"name":"The top 3 mistakes that make y&hellip;"}]},{"@type":"WebSite","@id":"https:\/\/conversion.com\/#website","url":"https:\/\/conversion.com\/","name":"Conversion","description":"Better results with evidence","publisher":{"@id":"https:\/\/conversion.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/conversion.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/conversion.com\/#organization","name":"Conversion","url":"https:\/\/conversion.com\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/","url":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","contentUrl":"https:\/\/conversion.com\/wp-content\/uploads\/2022\/12\/Colour-Logo.png","width":1821,"height":193,"caption":"Conversion"},"image":{"@id":"https:\/\/conversion.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/conversion.com\/#\/schema\/person\/843da25bf5d8480f182badf116cc1b4d","name":"James Flory","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/conversion.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/30287cbe25c2d5bb42ba74b00f2223af?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/30287cbe25c2d5bb42ba74b00f2223af?s=96&d=mm&r=g","caption":"James Flory"},"url":"https:\/\/conversion.com\/blog\/author\/james-flory\/"}]}},"_links":{"self":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/4530"}],"collection":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/users\/62"}],"replies":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/comments?post=4530"}],"version-history":[{"count":7,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/4530\/revisions"}],"predecessor-version":[{"id":7420,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/posts\/4530\/revisions\/7420"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media\/4696"}],"wp:attachment":[{"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/media?parent=4530"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/conversion.com\/wp-json\/wp\/v2\/categories?post=4530"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}