How to analyse and evaluate conversion marketing A/B tests, report on results and recommend next steps to your team/client.
When doing A/B tests is important to follow a set of steps:
Step 1. Identify the leaks in the conversion funnel using a tool like Google Analytics, Kissmetrics or Hot Jar.
When prioritising, target pages that have both high traffic and high impact on revenue. E.g. an enterprise landing page that receives 700 daily visitors.
Step 2. Do qualitative research to understand what is preventing users from converting.
This includes surveys, user interviews, talking with sales, customer support, using heatmaps and other tools. Use also learnings from previous experiments.
Step 3. Create an experiment hypothesis:
“For[Marketing Mary] visiting the[Home Page], we believe changing[funnel tool graphic] into a[interactive & linked graphic] will[increase home-> product page by 5%]
We believe this to be true because[research or previous experiment]”
It’s important to test one element at a time(e.g. the headline, the copy in one page or a picture) or you won’t know what made the test work(or didn’t).
When testing for copy use copy that is actually different(e.g.“download” vs“get this” or“get more conversions” vs“reduce your CPC”).
Also, don’t run multiple tests within the same funnel because that makes it hard to know where conversions are coming from.
Step 4. Create a variation per that hypothesis and test it against the existing version, using tools like VWO or Optimizely.
Make sure you split the traffic evenly.
Test during times when traffic is normal. If you’re an e-commerce site for example, don’t test during Christmas. Yes, you’d get results fast. But, people during Christmas don’t shop the way they normally do so, results are likely to be misleading. When you run an A/B test you want the traffic to be as representative as possible of your average traffic. So, look at Google Analytics and see if historically there are any spikes in traffic during certain times. And, talk to your marketing team and make sure there are no promotions or massive campaigns going on during the period of the A/B testing to make sure traffic is steady.
Other technical details: 1. If you're running an A/B test that redirects users from the original URL to a variation URL, use a 302(temporary) redirect, not a 301(permanent) redirect. This tells search engines that this redirect is temporary - it will only be in place as long as you're running the experiment. 2. No cloaking. Make sure that you're not deciding whether to serve the test, or which content variant to serve, based on user-agent
Step 5. Analyze the A/B test results, and see which variation delivered the highest conversions.
If there’s a clear winner you implement it, if there isn’t, you build new hypothesis and test again(NB: you start a new test from scratch. You don’t introduce an additional variable to a test that has already started or that variable won’t reach statistical significance most likely).
The clear winner must have 95% of statistical significance or above. If not, you can’t stop the test and call a winner, even if one variation is beating the other one by far. You can’t be sure about what the winner variation is until when you have statistical significance.
If you see small improvements in CRO and you’ve been seeing these small improvements for a while you may have reached a local maxima. In this case, it’s better to either redesign the page from scratch(with the risk that the new variation will lose) or to just focus on improving other pages. You can set the minimum % lift you want to see beforehand so every improvement smaller than that will automatically make the variation lose. If re-designing the site from scratch, make it as simple as possible, so you can then easily adjust it and add elements over time depending on the results of the experiments(following the growth driven design methodology).
Whatever the results, just trust the data and not your intuition. Even a small increase in conversion rate is something so just trust the test. And, small wins over time can add up to big results.
Judge the test according to the right metrics. If your goal is to increase profits, make sure that when conversions are increasing profit is also increasing. For example, showing a lower price means most likely more conversions but may also mean less profits.
Step 6. Share the results of the test and your learnings with the rest of the company.
Step 7. Move to the next hypothesis and experiment.
If you see a great improvement in conversion rate, you may think you’ve found the key thing that works for your market and you can just apply it all over your site. But you need to test first. So, you can take the learnings and design new experiments but just make sure you run the experiments before implementing the changes.
Step 1. Identify the leaks in the conversion funnel using a tool like Google Analytics, Kissmetrics or Hot Jar.
Step 2. Do qualitative research to understand what is preventing users from converting.
Step 3. Create an experiment hypothesis:
Step 4. Create a variation per that hypothesis and test it against the existing version, using tools like VWO or Optimizely.
Step 5. Analyze the A/B test results, and see which variation delivered the highest conversions.
Step 6. Share the results of the test and your learnings with the rest of the company.
Step 7. Move to the next hypothesis and experiment.
References: