The Rules of A/B Testing by Tyler Durden

1st Rule: You don’t talk about A/B Testing.

Don’t tell your visitors they are part of an A/B test. It would contaminate the test and would make them feel awkward. Try to hide the test as much as you can. If you have redirects or URL parameters showing up, make them not too obvious to the visitors.

2nd Rule: You DO NOT talk about A/B Testing.

Stop talking and start testing. Way too much time is spent discussing what to test and how to test and not enough time is actually spent testing stuff. Start small and iterate. Every minute without testing is a lost opportunity to learn something.

3rd Rule: If someone says “stop” or goes limp, taps out the test is over.

If you are testing on a large environment and someone reports an issue somehow related to your test, you stop it immediately. You fix the problem and then you start a new test. Tests need to be seamless, they should not disrupt the daily business.

4th Rule: Only two variations to a test.

If you’re not a master at testing, stick to a simple A/B test. Multi-variant tests are very difficult to master and require much longer to be statistically significant.

5th Rule: One test at a time.

Make sure you are running isolated tests. Keep them simple and iterate fast.

6th Rule: No header, no footer.

A/B testing works best if you limit your scope to simple elements of design or content. A/B test your copy, your images, your buttons. Only when you are a master try with complete user flows and bigger elements.

7th Rule: Tests will go on as long as they have to.

Significance, significance, significance. Keep your tests running as long as you have a statistical significance. Don’t cut corners and don’t cut tests short. It’s a good rule to keep tests running for a whole week, at least, in order to avoid short term seasonality in users’ behaviors.

8th Rule: If this is your first approach at A/B Testing, you HAVE to test stuff.

The only way to get good tests is to test often and fail quickly. Don’t spend too much time refining things at the very beginning, work on low hanging fruits and then work on the details, but definitely go and test NOW!

More about Tyler Durden

Published by Luca Sartoni

I lead three teams of engineers across four continents at Automattic. I love hiking, photography, and 3D printers.

12 thoughts on “The Rules of A/B Testing by Tyler Durden

  1. Good post as A/B testing is still not properly understood!
    Few points to consider –
    1) Try small tests in start – change in color of button, font size, less content vs more content, call to action vs two different page designs
    2) Have a clear goal – lots of companies don’t have a clear goal like people who reach URL Y or, % of people who added product to cart when I increase size of button

Leave a Reply

%d bloggers like this: