In a perfect world, all A/B and Multi-variate Tests you conduct should be as expectantly observed (and ghoulishly revisited) as the one conducted last week, pictured above.
Before the test launch of SpaceX Starship, Elon Musk said there was a one-in-three chance it would not survive to a successful landing. He was wise to set expectations. It exploded upon contact with the earth.
How do you handle and publicize your failed tests? I hope you celebrate them for the data you gathered. Put yourself in Musk’s shoes: He paid dearly for a pile of burned shards, but in return he got measurements throughout the flight that will help him rocket above the competition (sorry) to claim the profits in manned space flight he is chasing.
The only true failed experiment is the one where the data gathered is not applied to future efforts.
When you’re trying to optimize the profits of your business, most web metrics are unhelpful at least, and deceptive at worst. But what about the world of email marketing ? Kevin Hillstrom, in his excellent Mine That Data blog, gave this example to illustrate how conventional email metrics look at the wrong things:
Say you have a list of 500,000 e-mail addresses. You send your standard campaign on a Monday. Later in the week, you tabulate your results:
500,000 recipients
20% open rate = 100,000
Of the opens, 20% click through to the website = 20,000 visit website
Of the clicks, 5% convert and buy something = 1,000 orders
He compares these finding with what you’d get if you did something called a mail/holdhout test. You compare a control group that does not get the email with a test group that does. For instance, he suggests this breakout:
Mailed Group = 400,000 Recipients, $300,000 spent = $0.75 per customer
Holdout Group = 100,000 Held Out, $45,000 spent = $0.45 per customer
Incremental Lift = $0.75 – $0.45 = $0.30 per customer
Much more insightful!
This is why I’m not a fan of open/click/conversion. A mail/holdout test proves the actual value of an e-mail marketing campaign. In this case, we observe $0.30 lift, whereas open/click/conversion yields $0.20 lift. E-mail marketers, why would you not want to know that your campaigns are working 50% better than when measured via opens/click/conversion?
Kevin goes on to provide other interesting observations from his years of doing this sort of testing. You can’t go wrong by following his blog, and trying his approach to data-driven online marketing.
Everyone involved in technology and marketing has had this conversation: They’re in a social setting, and the subject comes up about how technology is crafting messages to match consumer behavior. Someone pipes in, “Oh, like in Minority Report? That’s so wrong!” Although Gmail’s ads are customized based on the content of email messages, I’ve noticed that few object to that practice. Instead, they’ll complain about the ads appearing in Facebook — ads that clearly are using information users have given the network about themselves.
It’s a Catch 22 for online marketers. In order to boost response rates, we need to know more about the people viewing our ads. This work of behavioral targeting sounds like a win/win: “We’ll only provide you with the ads that you will likely care about.” But in practice, consumers get spooked.I’m reminded of direct response research done years ago. It was a survey to find out how people like to be hit up for contributions to non-profit causes.
Here’s how consumers responded:
Least favorite: Personal asks
2nd Least: Telemarketing calls
3rd Least: Direct mail
What researchers at the time found telling was the direct correlation that existed between disliking a method of asking for donations and its effectiveness in getting them. In other words, personal asks — your sister-in-law selling Girl Scout cookies for her daughter — are most effective in terms of closing rates. The closing rates of telemarketing (back when this was a more viable medium for fund raising) were nearly as good. And a distant third in terms of effectiveness was direct mail.
So what’s really going on here? The accepted theory is this: We all have limited money to contribute to causes, and we would prefer to put off making decisions about where that money should go. Therefore, the most effective ways of forcing a decision are the least preferred.
Similarly, we love DVRs, because they allow us to zoom past commercials. They give us a way to avoid participating in commerce. They can’t touch us because we’re averting our eyes.
Behavioral targeting, if done properly, presents ads that also touch us. Thus, we look for reasons to hate the practice. Privacy is as good a reason as any and better than most!
So what’s the solution?
I do think that attitudes are slowly changing, and this change will eliminate privacy concerns as a reason to hate behavioral targeting.
Here’s an example. Consumers using social media are getting more comfortable with the various personas that they present on networks. They’ll show their “all-business” persona on LinkedIn, and their more casual facade on Facebook. Both are true depictions of the user, but they’re single facets of a full personality. Context determines how consumers behave on these sites. They are becoming accustomed to being watched by friends and business associates.
Reading Online Body Language
These same consumers are seeing how they can watch their friends right back. They are learning to “read” a friend’s feelings and preferences, based on online behavior. Consumers are getting accustomed to the online equivalent of body language. Or maybe that’s too strong a word. We’re all quite conscious of what we’re conveying, and body language implies unconscious action. Maybe what we’re doing on social media is more akin to a Kabuki dance.
Reading these dances is what marketers behind behavioral targeting are learning at a mass level, and turning into surprisingly accurate algorithms. But when it’s done by machines, in the service of a sale, many consumers still insist it’s “so wrong.”
Perhaps not forever.
To predict how people will react in the future — especially in areas such as privacy — I try to look at real world analogies. It’s not hard to imagine a web site as similar to a retail store. A while ago I wrote about how stores are analyzing shoppers using arrays of cameras. One system, called PRISM, is finding some unexpected insights into shopper preferences and behaviors. Shoppers aren’t getting freaked out. The prime reason: Most are oblivious to the surveillance.
But what if the people behind the cameras sprung up from their chairs and raced into the aisles, to say things like, “I saw you were looking at those lawn rakes. Can we interest you in some yard waste trash bags as well?” There would probably be lost sales, at the very least!
In the online world there are intercepts like this, but they are automated. This automated “intercept” is something people will become more accustomed to over time, as a generation weaned on social media, and used to their online movements being watched, comes into the majority. They will be able to understand that their behavior is being measured by machines as well as their online friends. They’ll realize there is no man behind the online ad “curtain” … just a predictive model.
Blurring The Lens To Reduce The Creep Factor
Another trend that will help the acceptance of behavioral targeting is a move toward more explicit boundaries. For instance, I expect an eventual backlash to camera surveillance such as PRISM. But before this reaction can take hold, the boundaries of store monitoring will likely improve. This improvement in technology will, for once, be toward discreetly blurring the “analytical lens” — instead of making it sharper.
Radio tomographic imaging is a new way to study store traffic. It uses arrays of extremely inexpensive radio transmitters and receivers, placed around buildings, to display people moving within. Below is a demonstration of the technology.
The benefits of this technology over cameras, at least to marketers, is cost — both in equipment and the labor of monitoring. Because people become moving “blobs” of color, it’s easier for computers to analyze traffic patterns and behaviors. Less can sometimes be more. Combine this information with RFID signals and you have a way to track a shopping basket all the way to check-out.
Imagine how this could be used:
Before he leaves that store, a consumer might someday pass an “intelligent end cap.” This smart store display knows — based on radio tomographic imagery, enhanced by RFID data sent to it — when to light up. The end cap would know the consumer has a yard rake, and offers him the trash bags he forgot to add to his list.
This hypothetical consumer will probably be grateful, since he really did need the bags for his yard project. And after all, he was just a blob of color moving through the store.
Ironically, this is the level of detail being used for most online behavioral targeting. This is the “privacy invasion” that is causing such a fuss on Facebook and elsewhere.
Over time, consumers will become more comfortable with behavioral targeting’s perceived betrayal of privacy.
That will leave only one valid objection the technique: They just don’t want to buy more stuff.
Online newsroom specialists iPressroom recently surveyed businesses to see what sorts of skills they are looking for in their marketing and PR pros. The survey had a small sample size, as many of these do, and this report’s many charts read far more into the results than can be reliably concluded. But I credit its authors for noting something that jumped out at me as well:
Rather than focus on attracting or pulling visitors to their website by publishing high quality content and researching popular language, organizations appear to be more interested with pushing out messages to “friends” through social media, even though, in many cases, those messages include hyperlinks back to their own websites. Until these organizations learn to develop a more sophisticated approach to building and managing landing pages and web content management on their websites, they will have difficulty evaluating their return on investment for these emerging channels.
(Emphasis mine.)
I found this report by reading a trendy headline somewhere. It proclaimed that marketing and especially PR executives are expected to possess skills that most are still scrambling to master. Here is a sample chart showing the data behind this assertion:
The employers surveyed should be commended for understanding the pressing demands of social media. However, they’re overlooking an equally important skill in their communications hiring checklist. They must hire people who understand the importance of good site content and how to measure its value. This is essential to making long-term gains from social media and search engine efforts.
It’s not enough to know how to attract eyeballs. The owners of those eyeballs had better find something on a site that’s worth experiencing and sharing.
A friend with a successful b-to-b eCommerce business posed a simple question to me: “If you could only do one or two things for an ecommerce business (that sells actual products rather than a service or software or something) to increase customer retention, what would you recommend?” Here are my recommendations, in priority order. What are yours?
Place your web address, with a compelling call-to-action, directly on the products being shipped. Make this call-to-action as time-sensitive as possible. Don’t be lame and do include a deadline. NO: “Fill out our warranty card online.” YES: “Set up an email reminder on our site so you’ll never forget to replenish. Do it by [date] and we’ll give you an automatic 10% off your next purchase, and free shipping!” Enclose a card reiterating the offer. This may be your last best shot at creating a repeat customer.
Follow up your shipment with a “We’d like to know if your products are fitting your needs” email or letter. Include a customer satisfaction survey that rewards them with something they can use with an immediate order. If you’re using snail mail, naturally you should enclose a printed catalog. Draw their attention to related items that can be found within it (or if it’s an email, found on the eCommerce site). If possible make the effort self-financing by generating an immediate re-purchase. Use the Net Promoter Score (NPS)* methodology in the satisfaction survey, to track current loyalty for this customer and as a way to track overall likelihood to repurchase as a trend over time.
Those are my recommendations. What are yours? Comments are especially welcome, for me and my friend.
*I’m including this because, although NPS has fallen out of favor as a predictor of company growth, and in other ways is definitely not perfect, I agree with Dale Wolf in that I like its simplicity. You need to use something as a predictive baseline that can (hopefully) be compared with real loyalty measurements. The NPS methodology, associated with Satmetrix Systems, Inc. and Fred Reichheld, is good enough to do the job.
It’s a Catch 22 for online marketers. In order to boost response rates, we need to know more about the people viewing our ads. This work of behavioral targeting sounds like a win/win: “We’ll only provide you with the ads that you will likely care about.” But in practice, consumers get spooked.I’m reminded of direct response research done years ago. It was a survey to find out how people like to be hit up for contributions to non-profit causes.