Email A/B Testing: What to Test, How to Test It, and What to Do With the Results

The A/B Testing Hierarchy

Not all A/B tests are equal. Some test things that move metrics by 1–2%. Some test things that move metrics by 20–40%. The hierarchy of impact:

High impact (test these first):

• —Subject line (affects open rate — everything downstream depends on opens)
• —From name (surprisingly powerful — "Santosh from SP Marketing" vs "Santosh Paudel" can shift opens 10–15%)
• —Primary CTA copy and placement

Medium impact:

• —Email length (short vs. long)
• —Send time and day
• —Personalization tokens

Low impact (not worth testing early):

• —Email color schemes
• —Image vs. no image
• —Specific word choices in body copy

Start at the top and work down. Testing button colors before you have optimized your subject lines is like repainting a car with a broken engine.

How to Run a Valid Email A/B Test

Sample size: You need at least 200 recipients per variant to get statistically significant results. Tests on small lists (under 500) produce unreliable data.

One variable at a time: Test subject line A vs. subject line B, not subject line A vs. subject line B with different body copy. You will not know what caused the difference.

Wait for full results: Let the test run to completion before declaring a winner. Typically 24–48 hours after send.

Document everything: Keep a running log of what you tested, what won, and by how much. Over 12 months, you will have a clear picture of what resonates with your specific audience — more valuable than any published benchmark.