The Rules of A/B Testing by Tyler Durden

1st Rule: You don’t talk about A/B Testing.

Don’t tell your visitors they are part of an A/B test. It would contaminate the test and would make them feel awkward. Try to hide the test as much as you can. If you have redirects or URL parameters showing up, make them not too obvious to the visitors.

2nd Rule: You DO NOT talk about A/B Testing.

Stop talking and start testing. Way too much time is spent discussing what to test and how to test and not enough time is actually spent testing stuff. Start small and iterate. Every minute without testing is a lost opportunity to learn something.

3rd Rule: If someone says “stop” or goes limp, taps out the test is over.

If you are testing on a large environment and someone reports an issue somehow related to your test, you stop it immediately. You fix the problem and then you start a new test. Tests need to be seamless, they should not disrupt the daily business.

4th Rule: Only two variations to a test.

If you’re not a master at testing, stick to a simple A/B test. Multi-variant tests are very difficult to master and require much longer to be statistically significant.

5th Rule: One test at a time.

Make sure you are running isolated tests. Keep them simple and iterate fast.

6th Rule: No header, no footer.

A/B testing works best if you limit your scope to simple elements of design or content. A/B test your copy, your images, your buttons. Only when you are a master try with complete user flows and bigger elements.

7th Rule: Tests will go on as long as they have to.

Significance, significance, significance. Keep your tests running as long as you have a statistical significance. Don’t cut corners and don’t cut tests short. It’s a good rule to keep tests running for a whole week, at least, in order to avoid short term seasonality in users’ behaviors.

8th Rule: If this is your first approach at A/B Testing, you HAVE to test stuff.

The only way to get good tests is to test often and fail quickly. Don’t spend too much time refining things at the very beginning, work on low hanging fruits and then work on the details, but definitely go and test NOW!

More about Tyler Durden

10 thoughts on “The Rules of A/B Testing by Tyler Durden

  1. […] That’s the beauty of A/B testing. A feature request in our bug tracker is a stone being added to a wall with mortar: it becomes a seemingly permanent piece of your architecture, of your user interface and of your responsibility. But an A/B test, why, that’s just a test! We have made no commitments to keeping it around. We’re not even sure we like it! Implementation only took a few hours so we could rip it out at any time for any reason and respond, “It was just a test.” Simply doing it, rather than talking about it, is rule #2 of A/B testing fight club. […]

    Like

  2. Good post as A/B testing is still not properly understood!
    Few points to consider –
    1) Try small tests in start – change in color of button, font size, less content vs more content, call to action vs two different page designs
    2) Have a clear goal – lots of companies don’t have a clear goal like people who reach URL Y or, % of people who added product to cart when I increase size of button

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s