TL;DR

  • Tracking via voucher codes underestimates campaign success. Tracking via responder matching, meaning total conversions, overestimates it.

  • The true effect lies somewhere in between. Control groups, also called hold-outs, make it visible.

  • With a clearly defined hold-out group, you measure the incremental lift, meaning the share of purchases that were truly triggered by print.

  • PostPal automates the complete setup: create hold-out, exclude it, track it, analyze it.

  • This turns direct mail into a measurable performance channel.

Problem: why most brands evaluate direct-mail success incorrectly

Many merchants see only who purchased. Not why the purchase happened. The two common tracking methods are:

  • Voucher redemptions

  • Responder matching

Both methods are valid because they offer a good compromise between measurability, accuracy and effort. But neither reflects reality correctly.

Vouchers regularly underestimate the effect

  • Some buyers do not redeem the voucher but were activated by print.

  • Others would have purchased anyway, and the voucher merely closes the sale.

In inactive segments, for example winbacks, vouchers usually underestimate success. In very active segments, for example a second-buyer push, they tend to overvalue it.

Responders overestimate the effect

  • Every order from the target segment is attributed to print.

  • This creates the upper bound, often significantly too high.

Both measurement methods only approximate the true campaign result and show the typical problem of marketing attribution: what do I want to believe? A control group solves this problem and shows the true result.

The gold standard: control groups

A control group is a small, representative part of your target audience that does not receive the mailing. Everything else remains identical, for example prices, promotions, segment logic, measurement timing and season. The only difference: print or no print.

This answers three core questions:

  • Which purchases would have happened anyway?

  • Which were triggered by the mailing, independent of vouchers?

  • How large is the real economic effect?

Direct mail thus moves from “feels effective” to “cleanly measurable”.

Excursus: automatic control groups in PostPal

Until now, measuring with control groups was extremely labor-intensive, which is why mostly very large teams used them: high manual effort, Excel, segment maintenance and error-prone analyses.

In PostPal, you can now create automated control groups for every campaign:

  • Automatic split into mailing vs. hold-out

  • Cross-channel exclusion via Klaviyo is possible

  • Daily conversion tracking (Shopify, Klaviyo)

  • Automatic lift analysis

This finally makes incrementality measurement practical.

Sample calculation

Comparison of mailing performance with the control group

Target audience

  • 50,000 recipients

  • 10% hold-out = 5,000

  • Mailing group = 45,000

Campaign result

  • Responders: 3,740 (8.31%)

  • Vouchers: 1,485 (3.3%)

  • Control group: 4.78% (= natural conversion rate)

Incremental effect

  • Mailing group: 8.31%

  • Control group: 4.78%

  • Lift = +3.53% (= actual conversion rate of the mailing)

→ 45,000 * 3.53% = 1,589 additional buyers

For campaigns outside strong promotions, voucher redemptions are often a solid approximation of the control-group result, as in the example above. During intense promotional periods such as BFCM, however, the picture changes significantly: voucher redemption then regularly underestimates the actual campaign effect. The mailing activates the customer, but the checkout happens through another channel or another voucher that is being promoted more prominently at that moment. In exactly these situations, the gap between voucher attribution and true incremental uplift becomes especially clear.

Clean incrementality measurement

1. Define the segment

Segment and offer need to fit together. Clean segmentation is the biggest lever. We recommend segmenting based on the RFM model. Only if this step is right does the campaign make sense. Otherwise, the results will disappoint, no matter which measurement method you use.

With PostPal’s Klaviyo integration and the RFM analysis from the Shopify app, segmentation is very easy.

2. Determine the control group

Statistically relevant sizes:

  • Minimum: 1,000 to 2,000

  • Ideal: 5,000+

  • Share: 5 to 15%

Hold-outs that are too small create too much noise.

Hold-outs that are too large, above 30%, mean you unnecessarily give up conversion uplift.

3. Exclude the control group

PostPal ensures that:

  • the hold-out receives no print,

  • but is fully tracked.

4. Send the campaign

  • same segment logic

  • same timing, with control and mailing group starting simultaneously

  • no channel leak, meaning the control group truly receives no print

  • Important: control and mailing group are treated the same on all other channels.

5. Define the measurement window

Direct-mail response is longer than many think:

  • 50 to 60% of purchases within 21 days

  • The remaining 40 to 50% afterward

PostPal uses a 60-day tracking window.

6. Analysis

If the hold-out group cleanly reflects the mailing group, you can extrapolate it.

Then the rule is: incremental revenue = campaign revenue, extrapolated revenue of the hold-out group

That allows you to calculate: CR, AOV, ROAS, CPO

(Info: in our next iteration, we will automatically calculate and display the total result for you.)

7. Scaling

When you test the mailing for the first time, we recommend using a control group. Only then can you see whether the campaign is truly incrementally profitable.

Once the setup is validated, you can contact 100% of the target audience. For existing automated campaigns, an annual control-group test is enough to ensure the effect remains stable.

When you do not need a control group:

  • when the mailing has already been validated several times

  • when the segment is very small and the lift would be difficult to prove anyway

When a control group is especially useful:

  • new campaign ideas

  • new incentives

  • new segments

  • changed market conditions, for example discount level, demand or advertising competition

Long story short: test when new. Send fully when proven. Verify annually.

Conclusion

  • Control groups make print measurable and economically assessable.

  • They show the true ROI between the coupon lower bound and the responder upper bound.

  • With automated hold-outs, direct mail becomes a data-driven performance channel.

If you need help with setup or want to find out whether a control group is worthwhile for your campaign, feel free to get in touch.