When you’re trying to optimize the profits of your business, most web metrics are unhelpful at least, and deceptive at worst. But what about the world of email marketing ? Kevin Hillstrom, in his excellent Mine That Data blog, gave this example to illustrate how conventional email metrics look at the wrong things:
Say you have a list of 500,000 e-mail addresses. You send your standard campaign on a Monday. Later in the week, you tabulate your results:
- 500,000 recipients
- 20% open rate = 100,000
- Of the opens, 20% click through to the website = 20,000 visit website
- Of the clicks, 5% convert and buy something = 1,000 orders
- Average Order Value = $100
- Total Demand = 1,000 * $100 = $100,000
- Demand per Recipient = $100,000 / 500,000 = $0.20 [per customer]
He compares these finding with what you’d get if you did something called a mail/holdhout test. You compare a control group that does not get the email with a test group that does. For instance, he suggests this breakout:
- Mailed Group = 400,000 Recipients, $300,000 spent = $0.75 per customer
- Holdout Group = 100,000 Held Out, $45,000 spent = $0.45 per customer
- Incremental Lift = $0.75 – $0.45 = $0.30 per customer
Much more insightful!
This is why I’m not a fan of open/click/conversion. A mail/holdout test proves the actual value of an e-mail marketing campaign. In this case, we observe $0.30 lift, whereas open/click/conversion yields $0.20 lift. E-mail marketers, why would you not want to know that your campaigns are working 50% better than when measured via opens/click/conversion?
Kevin goes on to provide other interesting observations from his years of doing this sort of testing. You can’t go wrong by following his blog, and trying his approach to data-driven online marketing.