Run holiday tests

1. Run bandit tests that will show the highest value variation of your test, more often.

This means you’re reaping the rewards while you test. Test elements like your seasonal value proposition, product names and descriptions, promotion names and descriptions, and CTAs in PPC ads, landing pages, and emails.

2. Conduct the same A/B tests after the holiday season when traffic is regular, to check the validity of your results.

Your holiday traffic might differ enough from your regular traffic to make these test results useless for most of the year. For many businesses, holiday traffic is an anomaly. Their behavior cannot be generalized throughout the year.

3. Test elements that have a high impact, such as seasonal messaging, rather than micro-conversions and small gain elements.

Focus on elements like your seasonal value proposition, urgency messaging, offered merchandise, promoted products, return policy, and shipping guarantees, like a receive it by Christmas promise. According to Optimizely, after a related #OptiChat, “Experts say that A/B tests with the biggest impact for the holiday rush are the ones that bust any fear, uncertainty or doubt a shopper might have. Types of tests to avoid during the holidays are any radical, structural changes that may break the site or slow performance.”

4. Optimize your website before the holiday by encouraging new customer reviews, mapping customer behavior, improving seasonal value propositions, and building your mailing list.

Use a scroll and click mapping tool like Hotjar to see how your visitors behave pre-season and on peak traffic days, then compare the behavior. Check if the data matches up and if it can be used to make smarter design/UX decisions year-round. Ask your best customers to leave a review as online reviews are the leading holiday influencer. Focus on growing your mailing list throughout the year as the larger your list, the more potential revenue.

5. Conduct QA on your website before the holidays and test if your website can handle an increase in traffic.

Test that every element on your website works and directs users through the funnel as expected. Optimize and test the mobile version of your website to avoid losing quality traffic. Open the pages on multiple mobile operating systems to check for layout changes. Litmus reports that during the 2013 holiday season, 50% of email opens were on mobile and over 71% of subscribers delete emails that don’t look good on mobile. In 2014, Best Buy’s site famously went down on Black Friday. For over an hour, their site said: “We’re sorry. is currently unavailable. Check back soon.” Given Best Buy generated $11.5 billion during the 9-week holiday period the year before, it’s safe to say that hour on Black Friday was a costly mistake.

6. If you want your holiday testing to be more applicable to the rest of the year, create a segment for returning visitors, and use it for your holiday testing.

People who have purchased from you before and had a positive experience are likely to surface again around the holiday season. So, while you’ll see an influx of new visitors, you should also see more returning visitors, and these are more representative of your regular traffic. Note that their intent is much stronger and their behavior may be different from the rest of the year. You should still run the tests again post-season, but segmenting should bring you closer to valid data.

7. Conduct conversion research to identify friction points on your website that you can transform into seasonal opportunities.

What is the biggest point of friction? Shipping cost? Shipping time? How much are consumers willing to pay for your products? Do your visitors view your products as higher quality? Can you charge extra for that? Gather data like how your visitors responded to each promotion, whether they were more interested in a $20 coupon or 10% off, and the most effective placements on your site for advertisements of these promotions. Use data to hypothesize new seasonal promotions before running the tests.