Glossary: outreach
What is A/B testing in cold email?
A/B testing is a way of comparing two versions of one thing, such as a subject line, by sending each to a similar group and measuring which gets the better result. Only one element differs, so the difference in results can be traced to it.
One test, two versions
Version A
- The current email
- Goes to half the list
- Sets the baseline
Version B
- One element changed
- Goes to the other half
- Wins only on a clear gap
An A/B test has three parts.
Version A is the current email. It goes to half the list and sets the baseline. Version B has one element changed, goes to the other half and wins only on a clear gap. Three things make that result believable.
One variable changes.
Change the subject line or the opening or the call to action, never two at once. Otherwise you cannot tell which change mattered.
The two groups are alike.
Split one list at random so both halves have the same mix of industries, sizes and titles.
One measure decides.
Pick the measure before sending. For cold email that is reply rate, since replies are what lead to meetings.
Here is an A/B test run for an invented company.
Imagine Sorrelgate, an invented company that sells HR software. It splits 1,200 verified prospects at random into two groups of 600. Version A uses the subject line "Quick question". Version B uses "Onboarding at your new office". The email body is identical.
Version A gets 12 replies, a reply rate of 12 ÷ 600 × 100 = 2%. Version B gets 24 replies, a reply rate of 24 ÷ 600 × 100 = 4%. Version B becomes the new baseline, and next week Sorrelgate tests a new opening line against it.
Replies from 600 prospects for each Sorrelgate subject line
Run an A/B test in four steps.
Testing is a habit, not a project. One clean test a week adds up.
The four steps in order
- 1Pick one thing
- 2Split evenly
- 3Wait
- 4Keep the winner
- 1
Pick one thing to test.
Start with what the reader sees first: the subject line, then the opening line, then the call to action.
- 2
Split evenly and at random.
Send both versions on the same days to halves of the same list.
- 3
Wait for the full sequence to finish.
Judge the test when both groups have had every follow-up, not after one day.
- 4
Keep the winner and test again.
The winner becomes version A in the next test.
Prymatica A/B tests subject lines, angles and CTAs weekly and improves your campaign on live data, turning cold prospects into interested replies and booked meetings. See how Prymatica's cold email process works.
Teams make these mistakes with A/B testing.
Several things change at once.
A new subject, a new opening and a new offer in one test tell you nothing about any of them.
The groups are too small.
A gap of two replies between two small groups is chance. Use groups large enough for the gap to be clear.
Opens decide the winner.
A subject line can win opens and lose replies. Judge on the measure closest to a meeting.
These glossary terms sit next to A/B testing.
Questions
People ask these questions about A/B testing.
What should you A/B test in a cold email?
Test the subject line, the opening line, the angle of the offer and the call to action, one at a time. These are the parts a reader sees first and acts on.
How long should an A/B test run?
Until both groups have received the whole sequence and replies have stopped arriving. Ending a test after the first email misses the replies that follow-ups bring in.
How many contacts does an A/B test need?
Enough that the gap between the two versions is clearly more than chance. The smaller the difference you are trying to detect, the more contacts each group needs.
What is the difference between A/B testing and multivariate testing?
An A/B test compares two versions that differ in one element. A multivariate test changes several elements at once and compares every combination, which needs a much larger list to give a clear answer.
Want your inbox filled with qualified leads?
Let's scale together. Book a free intro call and we'll show you exactly how we'll fill your pipeline with meetings from the industries you want to target.
No contracts. Start free. Cancel anytime.