The Humble but Effective A/B Test

Over the next few weeks, I’ll be using my blog posts to highlight the Test & Target tools you have at your disposal, as well as call out the most effective ways to use them.

I’ll start the series with the A/B test, one of the most fundamental tools in your arsenal. With multivariate tests getting the lion’s share of attention in the press, the effectiveness of an A/B test is often overlooked. That’s too bad, since an A/B test is often the baseway to start testing.

First, let’s define some terms. For our purposes, I’ll define an A/B test as a live test in which you measure the performance of two or more pages against each other. Examples would be testing two versions of your site’s homepage, or testing three versions of a particular landing page.

My OTTO Digital clients are often surprised when I suggest starting with an A/B test. But once I explain my reasoning, they quickly sign up with the plan. Here is why I recommend starting with an A/B test:

An A/B test encourages you to test radical alternatives

Unlike multivariate tests, which are designed to rigorously test specific elements on a page, an A/B test encourages you to try totally new page designs. You should take advantage of this fact and test out ideas that differ significantly from your control. At OTTO Digital we often tell clients that if you tack your tested alternatives up on a wall, you should be able to tell them apart from across the room.

An A/B test can gives you fast results

With an A/B test, you divide traffic into just a few buckets. If you’ve created recipes with dramatic differences, you’re likely to see a recipe or two emerge as a winner more quickly than you would with a more complex multivariate test. For getting results fast, nothing beats an A/B test.

A/B tests are simple to design and interpret

This is an important factor to consider, especially if site optimization is a new discipline at your company. Few things set back a nascent testing effort more than an exciting test that is later revealed to be flawed in some way. An A/B test provides a straightforward setup, with straightforward results.

An A/B test helps you focus your future MVT efforts

Once you run an A/B test and know which recipe is performing the best, you can further optimize that winner with a round of rigorous MVT testing. You’ll now be testing off an improved baseline. A/B testing is unmatched in its ability to reveal a winning concept and guide your future efforts.

So there you have it. Don’t hesitate to run the humble but effective A/B test. It’s often the best first step on your road to site optimization.

• John Broady
• 3 responses
•  Testing and Targeting

• Share on   Digg Reddit Facebook MySpace StumbleUpon Propeller Slashdot Newsvine ma.gnolia
• Bookmark at   del.icio.us Technorati Google Bookmarks Yahoo Bookmarks Yahoo! My Web Windows Live Furl