Sending one version of a direct mail campaign to every recipient may be simple, but it does not tell you which parts of the campaign actually drive results.
A/B testing replaces assumptions with evidence. By changing one element, measuring the outcome, and applying what you learn, you can improve future campaigns without redesigning your entire direct mail program.
Start with a clear testing hypothesis
Every test should begin with a specific prediction.
“Version B will perform better” is not enough. A useful hypothesis explains what will change, which audience will see it, and which outcome you expect it to influence.
For example:
Showing the offer as a percentage discount will produce more first purchases than presenting the same value as a dollar amount.
Choose the primary metric before sending. Depending on the campaign, that could be:
- Response rate
- Conversion rate
- Cost per acquisition
- Revenue per recipient
- Average order value
- Appointments booked
- Accounts activated
Your primary KPI should match the campaign goal. The direct mail metrics you select determine what counts as a winning result.
You also need a way to attribute each response to the correct version. Dynamic QR codes, personalized URLs, campaign-specific offer codes, and dedicated phone numbers can connect offline mail to measurable customer actions.
Six direct mail A/B testing strategies
The strongest tests focus on decisions that could meaningfully change customer behavior. Start with one of these six areas.
1. Test how you frame the offer
Two offers can have similar financial value but feel different to the recipient.
You might compare:
- A percentage discount against a dollar discount
- Free shipping against a product discount
- A free consultation against a promotional price
- An immediate benefit against a future reward
- A shorter deadline against a longer response window
Keep the eligibility rules, audience, creative, and CTA consistent. If the offers also have different terms, you may not know whether the framing or the underlying value drove the result.
Measure the final business outcome, not just initial interest. An offer that generates more scans but fewer profitable customers may not be the better choice.
2. Test the call to action
A strong call to action tells the recipient what to do and what they will receive.
Test meaningful differences in clarity, specificity, or effort. For example:
- “Start your trial” versus “Start your free trial”
- “Schedule an appointment” versus “Choose an appointment time”
- “Scan to learn more” versus “Scan to see your personalized offer”
- “Request a quote” versus “Get your estimate”
Do not test two CTAs that lead to different destinations or require different levels of effort unless that is the purpose of the experiment.
Each version should have its own trackable response path. Dynamic QR code tracking can distinguish engagement by creative, offer, segment, or campaign version.
3. Test the level of personalization
Personalization can range from including a recipient’s name to changing the entire message based on customer data.
Possible tests include:
- Personalized name versus no name
- Local imagery versus general imagery
- Product recommendations versus a standard product selection
- Customer-specific account details versus general messaging
- Lifecycle-based offers versus one offer for everyone
Test personalization that helps the recipient understand why the message is relevant. Adding personal information without improving the experience may create discomfort instead of engagement.
Make sure the data is accurate and appropriate for the use case. Automating personalized direct mail can make variations easier to produce, but the personalization strategy still needs a clear purpose.
4. Test creative or mail format
Creative tests can help you understand how design affects attention and action.
You might test:
- Photograph versus illustration
- Product-focused imagery versus lifestyle imagery
- Short copy versus more detailed copy
- Bold colors versus a more restrained palette
- Postcard versus letter
- Standard postcard versus larger postcard
Keep the difference focused. Comparing a bright postcard with a detailed letter changes the format, copy length, imagery, and visual style at once. Even if one performs better, you will not know why.
Format tests also need to account for cost. A more expensive format may still win if it produces enough additional revenue or higher-value customers.
5. Test audience or trigger criteria
The audience often influences performance as much as the creative.
You can test groups based on:
- Purchase history
- Customer lifecycle stage
- Geographic area
- Product interest
- Loyalty status
- Previous engagement
- Time since last purchase
- Account or lead value
Be careful when interpreting audience tests. Two different segments are not interchangeable randomized groups. If one performs better, that may reflect an existing difference between the audiences rather than the mailpiece itself.
Audience testing is most useful for deciding where to invest and which message fits each segment. Following targeted direct mail practices can also help prevent suppression, eligibility, or data-quality problems from affecting the test.
6. Test timing and journey placement
A relevant mailpiece can still underperform if it enters the customer journey too early or too late.
Test timing decisions such as:
- Sending soon after a customer event versus waiting
- Mailing before a digital follow-up versus after it
- Using mail as the first touchpoint versus an escalation channel
- Sending one reminder versus a timed sequence
- Triggering after one period of inactivity versus another
You cannot control the exact day each USPS piece arrives, so define timing tests around production dates, trigger rules, and reasonable delivery windows.
When direct mail is part of the customer journey, available production and postal events can help you coordinate the next email, SMS message, or sales follow-up.
A/B testing versus multivariate testing
A/B and multivariate tests answer different questions.
A/B testing
An A/B test compares two versions while holding other important variables constant. The audience should be divided randomly when possible so the groups are comparable.
A/B testing works well when you want a clear answer about one decision, such as:
- Which offer performs better?
- Which headline generates more responses?
- Does personalization improve conversion?
- Which CTA leads to more appointments?
Its main limitation is speed. If you want to evaluate several elements, you may need to run multiple tests.
Multivariate testing
Multivariate testing evaluates combinations of variables. You might test several headlines with several images to determine which pairing performs best.
This approach can reveal interactions between elements, but every additional variation divides the audience into smaller groups. Without enough recipients and responses in each group, the results may reflect random variation rather than a meaningful difference.
Use multivariate testing when you have sufficient volume and a specific reason to evaluate combinations. If your audience is limited or your campaign is new, start with A/B testing.
Lob’s dynamic templates and conditional content make it possible to create variations without rebuilding each mailpiece manually.
How to run a reliable direct mail test
Define the decision
Write down what you will do if version A wins, version B wins, or the result is inconclusive. If the result would not change a future decision, the test may not be worth running.
Change one primary variable
Keep the audience selection, send window, mail class, tracking method, and other creative elements consistent unless one of those factors is the variable being tested.
Assign recipients fairly
Randomly divide eligible recipients between versions whenever possible. Check that the groups do not contain obvious imbalances in geography, customer value, lifecycle stage, or previous engagement.
Set the measurement window
Give recipients enough time to receive the piece and complete the desired action. Do not stop a test as soon as one version moves ahead.
The appropriate window depends on the campaign, offer, delivery timing, and buying cycle.
Evaluate business impact
Compare response and conversion, but also review acquisition cost, revenue, customer value, and campaign expense.
A version that produces more responses may not be the winner if those responses do not turn into valuable outcomes.
Build testing into every campaign
One campaign result should inform the next campaign, not disappear into a presentation or spreadsheet.
Maintain a shared testing log that records:
- The hypothesis
- Audience and exclusions
- Versions tested
- Primary and secondary metrics
- Measurement window
- Result
- Decision
- Recommended follow-up test
Over time, this creates a useful record of how different audiences respond to offers, formats, personalization, and timing. It also prevents teams from repeatedly testing questions that have already been answered.
Do not treat one win as permanent. Customer preferences, pricing, competition, and market conditions change. Retest important assumptions when the surrounding conditions are meaningfully different.
Test and optimize direct mail with Lob
Lob helps teams create campaign variations, personalize mail with customer data, automate production, and connect responses to individual versions. Reusable templates make testing easier without requiring a complete redesign for every experiment.
Book a demo to see how Lob can support measurable direct mail testing at scale.
Frequently asked questions about direct mail testing
FAQs
What is A/B testing in direct mail?
Direct mail A/B testing compares two campaign versions to determine which performs better against a defined metric. A reliable test changes one primary element, assigns comparable recipients to each version, and tracks responses separately.
What should you test first?
Start with a decision that could materially affect performance, such as the audience, offer, or CTA. Small decorative changes may be easy to test but less likely to produce useful business insight.
How large should a direct mail test be?
There is no universal test size. The required audience depends on your expected response rate, the size of the difference you want to detect, and the level of statistical confidence you need.
Estimate the required sample before sending rather than deciding whether the result is meaningful afterward.
How long should a direct mail A/B test run?
Run the test long enough for both versions to move through production, postal processing, and the expected customer response window. Keep the window consistent across versions.
Can you test more than two direct mail versions?
Yes. Multivariate testing can compare several versions or combinations, but it requires more recipients. Make sure each variation has enough exposure and responses to support a useful conclusion.
Answered by:
