A/B Testing Done Right

A/B testing, a.k.a. “split testing,” is not as easy as Jackson 5’s lyric “As simple as…ABC” would have you believe. Sure, it sounds basic, but that doesn’t stop people from screwing it up and then swearing by their flawed results.

Are you testing yet? (If not, you should be.) Are you getting the most out of your tests? Read on for quick tips on A/B testing the right way.

Tip #1: Only One Variable, Please

“15 Items or Less” may work at Trader Joe’s, but it’s not good for testing. Granted, sometimes you want to study more than one variable, but unless you’re overhauling the site or rebranding, you shouldn’t do it.

The reason for my stern warning is that if you’ve tested subject line, photo, headline, and layout all at once, then how do you know which variable had the greatest impact?

This type of “too much of a good thing” experimentation also tends to lead to sweeping generalizations that all elements in the test are equally responsible for the variation’s success or failure. Obviously that’s untrue and it simply means you must test again, which begs the question: why not just do it right the first time?

Tip #2: Look at the Whole Funnel

I’m a big fan of whichtestwon.com, a cool site that offers a weekly A/B or multivariate test case. What makes the site cool is that it features a variety of tests (print, web, email) run by a wide cross-section of companies. What makes it slightly less exciting is that readers typically only see one statistic, which is good for a brief article, but bad for making business decisions.

To truly understand performance, you should be looking at a variety of numbers. Ideally your tracking codes will be monitored at every step of the process. If they’re not, well, maybe it’s time to change that.

Testing subject line and looking at open rate is a classic example of how marketers can take a myopic view. The subject line “Cheaper Than a Meal!” may get more people to open your message, but if your price point is too high, it may not generate as many orders as a version that’s upfront about your product’s cost.

Tip #3: Test Interesting Variables (and Test Them Again)

True, you can’t go wrong with a good subject line test, but don’t limit yourself to something so basic. You can also test things like photos, layout, benefit descriptions, and the size, shape and color of your call to action buttons.

Also don’t assume that just because something worked once or worked last year it will work again. Designs go in and out of fashion just like clothes, so it’s important to retest, lest you be caught wearing pastels when brights are all the rage.

Tip #4: Mix It Up With a Four-Way

Puns aside, a four-way test can help you get even more out of your testing. To conduct a four-way test, simply pick two independent variables and plan to test all of them. You can use a chart like this:

Variables 1 2
A Version 1A Version 2A
B Version 1B Version 2B

Four-ways are double the work of a basic A/B tests, and if you want accurate results, you’ll need a large audience. But if you’ve got the time and the audience, getting to test two independent variables at the same time – and getting reliable results – is pretty rad.

Tip #5: Dig Deeper for Answers

Probably the biggest mistake I’ve seen people make is to assume that the numbers tell the whole story. While data is essential, you may not have all of it. And when you dig deeper, a new story may reveal itself. Even though one email may have sold more widgets or garnered more downloads, if your tactics were shady, it may do long term damage to your brand. The lesser-performing message could also have unexpected value if it appeals to a different segment of your users.

Don’t feel boxed in by the statistics you have. Go with your gut and come up with new ways to test by segmenting your audience, soliciting feedback or simply coming up with new metrics to track.

What tips do you have when it comes to A/B testing? Share them below!

Bonus Links!

If you liked this, you might also enjoy…

Netflix’s Big Goof: 5 Copywriting Blunders in Its Recent Announcement
The Copywriter’s Secret Weapon
How to Write With Credibility


~ by SFcopywriter on September 27, 2011.

4 Responses to “A/B Testing Done Right”

  1. Great tips, I have seen some people are Testing just because they do not want to launch the email without the Test and they end up testing only the Subject Line, I am not sure if open rate is always a good indicator of which subject line did better, what do you think?

    • Thanks for the kudos, and it’s a good point that it’s not worth it to simply test for the sake of testing. You should only invest your energy and resources in the tests that you believe will have a notable impact.

      Despite what I argue above about looking at the whole funnel, I would say that open rate is generally a good indicator of which subject line did better. But the key word there is “generally.” I’ve personally seen a bunch of tests where the “winning” subject line was actually the loser when it came to conversion. It all depends on the subject lines, the content and your primary business goal (conversion, revenue, etc.).

      Good luck with your tests!

  2. Great post! So many people aren’t testing which is madness – and when they do, it is so often implemented poorly. Great tips for those needing some tutelage!

Share your two cents

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: