So Young Park, Director of eCommerce and Direct Marketing for A&E; and the History Channel, began running A/B tests on the site in 2003 -- an eon in this brave new world of scientific testing -- and she's hooked.
"We wanted to understand which merchandising spots, images, and messaging on the page were performing better," she says. "So that's why we started testing."
For example, an Easter promotion last year showed clearly that one message was performing 50% better than the other. She took down the one that was underperforming, "and that saved the promotion. We wouldn't have even known that something else could have performed better without testing."
Should other ecommerce companies try testing, if they aren't already? "Yes, definitely," Park says. "It's really important for improving and optimizing promotions as a campaign is running."
Park shared 6 tips for marketers entering the testing arena:
#1. Take analytics one step further
Web analytics highlight the performance of whole pages and particular areas of a site that get more clicks. "But it doesn't show what images or messaging performs better," Park says. Testing takes analytics to the next step.
#2. Have a clear idea of your goals
Some companies feel overwhelmed because the variables to test seem endless. "Go into it knowing what goals you want to reach," Park says. "Not giant goals that will take a year. Start small, especially if you've never done it before."
You might simply focus on running tests to help you choose a threshold for free shipping, for example. "We spent September testing different promotions and trying to figure out from the results what we could apply toward the holiday season," says Park. "We really hadn't done that before. In the past, we'd say, 'We want to do free shipping over $75 because we want to raise AOV and that level is still affordable.'"
With those test results, Park can be confident that her holiday offer will be successful.
#3. Be disciplined
You've heard it before: don't change more than one variable at a time. But marketers want instant gratification. "They'll say, 'We don't have a lot of time, so let's just squeeze this other thing in,'" Park says. "You really shouldn't do that."
That's because you need to be sure you can isolate the variable that's making the difference. In other words, don't change the color of your headline and the wording of your headline in the same test.
#4. Don't make an important decision based on a single test
If you want to discover whether your visitors respond best to illustrations or photos of your product, you might be tempted to run one A/B test and then use the results across the board.
Perhaps you find that a photo works best for specific product pages. But you may find that an illustration works better on a page with more copy. Dig deeper to see what lies further beneath the surface.
"There are so many variables—make sure you consider context, not just the test results in isolation—and then keep testing to confirm the initial results," Park says.
#5. Make sure your vendor or IT department can change things on the fly
Suppose you're testing a headline on a particular promotion: "Free shipping when you spend $100" vs. "Spend $100 and get free shipping."
If you run the test for a couple of weeks and see significant differences, you need to be able to implement the winner quickly. "You don't want to be in the position that one offer is clearly performing better than the other but you can't get it changed in time to affect sales," Park says.
#6. Testing doesn't have to be a full-time job
Don't let lack of personnel keep you from running tests. "Obviously, most companies can't have one person who's the tester," says Park. Testing might be one task among many that one or two people perform. Get some small wins that make a positive impact on your goals and then use that to make a case for getting a freelancer or full-time analyst.
At first, you may want testing to be a fun team project that multiple people get experience with. This will help with buy-in and support down the road.