In early 2020, I was given the task of running email marketing for a small startup. As I looked for ways to make the campaigns more effective, I turned to A/B testing. And after a bit of experimentation, I was sending emails that moved the needle toward business growth.
Each email marketing tool approaches A/B testing a little differently, but here, I'll walk you through how to do a Mailchimp A/B test.
Note: Mailchimp's Free plan doesn't include A/B testing, so you'll need to be on at least the Essentials plan. I'm using the Standard plan (one level above Essentials), but if you use the Premium plan, you'll have access to the more advanced multivariate testing feature as well.
How to create an A/B testing campaign with Mailchimp
A/B testing (or split testing) your email marketing lets you find out if your subject line is more effective with an emoji, which day of the week your emails get the most clicks, or whether a CTA button converts better than a text link.
Not only do A/B tests help you optimize your emails, but they also help you understand your audience's preferences. You'll know what time of day your audience likes reading their emails, whether they like watching videos or not, and which offers they're willing to spend money on.
Before running an A/B test, you should already have the email or drip campaign you want to test set up on Mailchimp. Once you have that, follow the steps below to run your split test.
1. Set up your campaign and select your audience
Log in to your dashboard, and click Create in the menu on the left side of your screen. Then click Email.
You'll see a dropdown menu—select A/B test. (If you're using the Premium plan, you'll see Multivariate test in place of A/B test.) This feature allows you to send up to three variations of an email to see which one resonates most with your audience.
Now enter your campaign name in the field, and click Begin.
Next, you'll have to choose who'll be receiving your email campaign. You can choose your entire audience or a segment of your subscribers.
If you don't want to send your email to all your subscribers and you don't have a segment yet, you can create a new segment. Mailchimp has pre-built segments based on the demographics of your subscribers or how much your audience is engaging with your emails. If you don't want to use Mailchimp's pre-built segments, create one from scratch.
2. Select your variables and settings
Next, you'll choose the variable you want to test. Mailchimp offers four variables for testing: Subject line, From name, Content, and Send time.
No matter the variable you choose, Mailchimp allows you to add up to three variations of the variable to test. When you click on a variable—say, subject lines—you can use the plus or minus buttons to add or remove variations.
Each variation will create a campaign combination (as you can see in the Summary section in the screenshot above). So if you're testing three subject lines, Mailchimp will create three combinations of your email. Personally, I like to test two variations at a time to keep things simple and easy to analyze.
Under the Variables section, you'll choose the percentage of your subscribers that will receive your test combinations. As you set the slider to the percentage you want, check the number of recipients that will receive your combinations in the Summary section.
No matter the variable you pick, ensure that you don't go lower than 10%. From personal experience, I'd say 50% is the sweet spot when you're testing two variables. This way, 25% of them get one variation each, and the remaining 50% get the winner. If you're testing Send time, however, send the emails to all the subscribers in your chosen audience or segment. 50% of them get the email at time A and the other half get the email at time B.
Finally, you'll choose how you want Mailchimp to decide the winner of the test and how long it'll take. Mailchimp offers four ways to determine a winner: Click rate, Open rate, Total revenue, and Manual selection.
If you're testing your Subject line, From name, or Send time, choose Open rate. If you're testing your email content, let Mailchimp choose a winner based on Click rate or Total revenue (if it's commercial content). If you'd like to pick the winner of the test yourself, pick Manual selection.
Then choose how long you want your test to run.
3. Set up the campaign
On the Setup page, you'll be able to specify your variations, depending on the variable you're testing. For example, I chose to test two different subject lines, so there are fields for me to put in both subject lines and a preview text.
There's also a section where you can specify your From name and From email address. Finally, there's the Tracking section where you can tick the boxes next to your preferred tracking options.
As you can see in the image above, Track opens and Track clicks are already checked and cannot be unchecked because they're used to determine the winner of the test. But if you want to activate Google Analytics link tracking, you can check the box (and uncheck it if you change your mind).
4. Add content for your campaign
Now it's time for the content. If you chose to test subject lines, from name, or send time, choose a layout from the Template section of the content builder.
When you click Next, you'll move to the Design section, where you can include the content of the email you want to test. When you're done adding the body of your email, test it to ensure that it's working as it should. Once you've sorted that out, click Confirm before sending.
However, if you chose Content as the variable you want to test, this step will look a bit different. Instead of choosing a layout and designing an email right away, you'll be presented with a preview for each variation.
For each variation, click Edit content or Add content. Choose your layout from the Template section, and design it. When you're done designing and testing it, click Next. This will take you to the Description tab, where you'll type in a short text to describe the content variation you just created. This will help you differentiate both variations when you're analyzing your results. For example, I described my content variations as Classic Welcome Email and Light-hearted Welcome Email.
Once you've done this, click Continue to Content. This will take you back to the preview page of your variations. Then click Next.
Note: When testing content variations, you have the freedom to try out different things. For example, you could try images only vs. text only, videos vs. infographics, social proof vs. no social proof, and static images vs. GIFs. You can also test out different offers and CTAs.
5. Confirm and send your A/B testing campaign
Clicking Next will take you to the Confirm page, where you'll be able to review your A/B testing combinations and send them once they're ready.
Before you send, review the pre-delivery checklist to resolve and edit any criteria that might prevent your combinations from delivering optimally. When you're done, click Send to send the variations out immediately or Schedule to send it out at a pre-selected date.
If you have a Manager user-level account with Mailchimp, you'll receive an email when a winner is selected.
6. Choose a winner manually
Remember when you chose how Mailchimp would decide the winning combination? Mailchimp provided these options: Click rate, Open rate, Total revenue, and Manual selection. If you chose Manual selection from the dropdown menu, you'll be able to assess the campaign reports after the tests are delivered and choose a winner yourself.
To choose a winner manually, navigate to your Mailchimp dashboard, and click Analytics in the left-hand menu, then click Reports. There, you'll see all the campaigns you've been working on. Find the A/B test campaign you want to review, and click View report.
Once you're done reviewing the results for each combination, navigate to the Finalize your test section, and tick the button next to the winning combination that you choose. Finally, click Send to Remaining Recipients to release the winner into the inboxes of your remaining subscribers.
Tips for running A/B tests on email campaigns
If you're new to A/B testing, here are some tips to ensure that you get the most out of your A/B testing campaigns:
1. Develop a hypothesis
To increase your chances of running effective A/B tests, you need to develop a hypothesis that reflects what you want to achieve with A/B testing. For example, your hypothesis could be that emails with your subscribers' names in the subject line will perform better than emails without them. Or you could guess that emails with images will perform better than text-only emails.
These working hypotheses can help you choose the variable you're going to test and what to look out for as you analyze your results.
2. Test the emails that matter
Instead of testing the "Happy Thanksgiving!" emails that you send only in November, test emails that you frequently send: welcome emails, newsletters, onboarding emails, automated follow-up campaigns, and promotional emails. You probably send dozens of these emails to your audience every month, so if you can increase the open or click rates even by a small margin, you'll get a better ROI on your email marketing efforts.
3. Test high-impact, low-effort variables
If you're just starting out with A/B testing, focus on testing variables that have the most impact and take less time to test. These include subject lines, CTA copy, headlines, and colors, among other things. Once you've optimized these elements on your emails, you can start testing high-effort elements like copy length, offers, and tone.
4. Test one variable at a time
Testing multiple variables simultaneously makes it difficult to understand results and figure out which variation is causing the changes. As you saw in the step-by-step breakdown, Mailchimp allows you to test up to three variations of a specific variable. To get the best results, send out your combinations to a large sample of people to get more accurate results.
5. Wait for a week to get the final results
When you send out email variants to your subscribers, give them enough time to engage with the emails before analyzing your results. The results you get after 24 hours will likely be different if you wait for up to seven days (and four to five at minimum).
Note: This point is up for debate. Mailchimp suggests that you run your A/B test for up to four hours, but I think that a week is an ideal time to run your test. The reason is simple: not everyone will engage with your email the second it arrives in their inbox. Some experts even recommend running a test for a few weeks before analyzing results. This approach works well if you're testing high-effort variables like offers and discounts. But if you're testing low-effort variables like subject lines and email layouts, a week is more than enough.
Run effective A/B tests with Mailchimp
A/B tests can be really confusing to set up, especially if you're just starting out with an app—or with A/B testing more broadly. Mailchimp has made the entire process as simple and straightforward as possible, but there are still some quirks.
If you find that the A/B testing feature is too simple for you, try upgrading to the Premium plan that allows you to run multivariate tests. Multivariate tests let you test multiple low- and high-effort variables to see how small changes to your emails can have a big impact on your engagement.
Related content:
