The Basics of A/B Testing Single Emails

2 minute read

Tim D'Avis Senior Digital Strategy Consultant

We all wish we did more testing, but we don’t. And when we do, we’re sometimes unsure of what the results mean. The data is just never as magical as we think it’s going to be – amiright?

Why is that? More often than not, we test small changes that are either inconsequential or only consequential at a massive scale. If we test the color of a green CTA vs a blue CTA, the results are likely to be inconclusive unless we’re sending to millions of people.

So how do we make tests that show meaningful results? We need to test bigger changes.
Use distinctly different approaches that are designed to challenge your assumptions. Let me show you what I’m talking about — here’s my recommended pattern for testing a new newsletter or email type.

First, test your ‘From Name’

Did you know From Names are the #1 factor in open rates? It’s true. Test a few distinct variants on your from name before you begin testing your subject lines. Test from a person, test from an organization, test with both (like “Tim at Secondstreet”). Look at your open rates, then pick a winner and stick with it.

Email subject lines on an mobile view

Second, test subject line formats

Content companies can’t be in the business of testing their newsletter subject lines constantly — it’s a huge resource drain. So try testing formats – content synopsis, individual headlines, dates, issue numbers. Test it all, and test it multiple times to make sure you’re getting a clear signal. Then, pick a winner based on open rate and move on.

Subject line formats

Finally, test your layouts

There’s a lot to test here, but be reasonable. Pick 3–4 layouts/approaches that you like and run tests for a week. Pick your winner based on click-to-open rate (divide unique clicks by the number of unique opens) so that you’re comparing things fairly.

Testing layouts mobile view

Test these three ways in order, methodically, over the course of 2–3 weeks. If you see a dip in engagement after a few months, repeat the second and third steps to see if you can gain back engagement.

Reliable products. Real results.

Every day, thousands of companies rely on Upland to get their jobs done simply and effectively. See how brands are putting Upland to work.

View Success Stories