The most important piece of education about A/B testing was released on the 12th of February 1993 in the form of a full featured movie, starring Bill Murray and Andie MacDowell. Of course I’m referring to the movie “Groundhog Day”.
The same structure has been successfully used by Doug Liman in “The Edge of Tomorrow” with Tom Cruise.
In both movies the main character finds himself living the same day over and over again, refining his strategies through trial and error, until he overcomes the adversities.
If you think about it, split testing is somehow the same: we try different options selecting the best performing ones under the same statistical conditions. Unlikely the previous examples we have to deal with scarcity of resources, the hard rules of physics and the harder rules of business.
With the growing fame of the term “Growth Hacking”, the availability of resources regarding split testing, CRO, usability testing has dramatically increased over the last few years. There are books, podcasts, webinars, articles, and entire publications on this topic. A simple search on Google for the term “split testing” presents almost nineteen million results.
How to cut through the mist and set up simple yet solid tests on our WordPress sites?
I plan, design, and implement split tests on a daily basis and over the years I refined a list of rules, easy to remember, yet essential for your first split tests.
How many of you are familiar with split tests? The idea is simple: two different experiences are offered to two statistically independent but comparable groups of people, and the results are compared with each other to determine which experience works best.
This can be done using several tools: Google Experiments, Optimizely, Kissmetrics, Unbounce, custom code. Today we are not going to focus on how technically perform split tests, but on what’s important to keep in mind, no matter the tool you are going to use.
Tyler Durden’s rules of A/B Testing
A few months back I came up with the eight rules of A/B testing, a direct adaptation of the eight rules of Fight Club.
1st Rule: You don’t talk about A/B Testing
Don’t tell your visitors they are part of an A/B test. It would contaminate the test and would make them feel awkward. Try to hide the test as much as you can. If you have redirects or URL parameters showing up, make them not too obvious to the visitors.
2nd Rule: You DO NOT talk about A/B Testing
Stop talking and start testing. Way too much time is spent discussing what to test and how to test and not enough time is actually spent testing stuff. Start small and iterate. Every minute without testing is a lost opportunity to learn something.
How to define what to test and how? Most times I’ve seen people testing on parts of their websites just because “it was obvious” they needed to be optimised. But the definition “too obvious” is not good enough for me. My advice is to map the current flows and gather enough data based on your experience. Then define the result you want to achieve. Then walk backward to identify actionable data. At this point what to work on will pop up by itself and you’ll find out it was not so obvious at the very beginning.
3rd Rule: If someone says “stop” or goes limp, taps out the test is over
If you are testing on a large environment and someone reports an issue somehow related to your test, you stop it immediately. You fix the problem and then you start a new test. Tests need to be seamless, they should not disrupt the daily business.
4th Rule: Only two variations to a test
If you’re not a master at testing, stick to a simple A/B test. Multi-variant tests are very difficult to master and require much longer to be statistically significant.
5th Rule: One test at a time
Make sure you are running isolated tests. Keep them simple and iterate fast.
6th Rule: No header, no footer
A/B testing works best if you limit your scope to simple elements of design or content. A/B test your copy, your images, your buttons. Only when you are a master try with complete user flows and bigger elements.
7th Rule: Tests will go on as long as they have to
Significance, significance, significance. Keep your tests running as long as you have a statistical significance. Don’t cut corners and don’t cut tests short. It’s a good rule to keep tests running for a whole week, at least, in order to avoid short term seasonality in users’ behaviours.
8th Rule: If this is your first approach at A/B Testing, you HAVE to test stuff
The only way to get good tests is to test often and fail quickly. Don’t spend too much time refining things at the very beginning, work on low hanging fruits and then work on the details, but definitely go and test NOW!
Usability tests: just three things to remember
Testers are not real users, no matter how good is the test
No matter how refined is the test you are conducting, be aware that a few elements are different from reality. Imagine you are testing a signup flow. Your goal is to reduce the number of people who give up before the end. Testers never give up. They are payed to test so they feel that giving up is forbidden. They will keep on exploring possibilities, and that is not necessarily what you are looking for in a usability test.
Testers are meant to find problems not to solve them
People are exceptionally good at giving advice. Especially if they don’t have to put the money where their mouth is. Let testers find problems but don’t listen to them if they try to suggest a solution. Your engineers are the right people who have to provide a solution, rely on them.
Believe in what they do, not in what they say
If testers say that your signup process is lean but they take double the time you expected to completed it, don’t listen to them. If they suggest to move a button around, look at what they do to find that button, not at where they say they would expect it.
A testing calendar is key
Better than investing time in trying to refine your tests to the bone, put together a sustainable testing calendar. As we said resources are limited and planning is key. Let’s say you can design, implement and review a test every month. Set up a calendar and stick to it. Say that you can do usability tests (online on person) every second Wednesday of the month. Good, go for it.
Just like Phil in the Groundhog Day, mastering tests and achieve results require experience. When you have the basics of statistics, a general understanding of the classic pitfalls, you are good to go. Don’t hesitate, every minute without testing is a missed opportunity.
Picture courtesy: Ivan Gatic