How do you play poker?
I recently sat down at a friendly poker game with a bunch of my neighbors. It turned out one of them had never played before. We walked him through all the rules, including that he started with two cards, called a ‘hand,’ and ended with five cards. He began playing and after a bit he asked a great question, “What is a good starting hand?”
"Ensure that you have a sound result, without bias, with significance, before you go to your boss asking for an investment"
This question turned out to be harder to answer than I had expected. After a while I realized that if you didn’t know how to play the game, you didn’t know when to fold, when to bet big and when to wait-and-see how things turned out, the game was extremely difficult. It required developing a hypothesis, testing it, reading the results correctly, and then defining and executing your strategy. The same is true for marketing analytics.
When the subject line contains the word FREE people are more likely to convert.
The definition of the hypothesis is potentially the most crucial step in marketing analytics. That hypothesis needs to be testable. Which means that you must take into consideration what your operations team can really test and what data you can access. There are many versions of a hypothesis as well. For example, “If we use the word FREE on the second email of a three-part, multi-channel, multi-touch campaign then people are more likely to convert.” Well, maybe, but can you think how challenging that test will be to execute? and then act on regularly? Keep the hypothesis, simple, germane, and easy to do.
Control the variable(s)
It is easy to confuse a lot of hypotheses within a single test. “Maybe if we use the word FREE with our best customers they will not unsubscribe, so let’s make them into a different segment.” Well, now you are not testing just the subject line, but also the segmentation. You need to keep as many things constant as you can – the message, the creative, the segmentation, the channel, etc. Only change what you want to test. Also, make sure that when you are testing email, someone else isn’t testing a landing pages. Otherwise, that great subject line you came up with, may go to landing page that simply doesn’t perform. This will lead you to, incorrectly, conclude your email just didn’t have the right stuff. Control, or at least coordinate, all the variables, not only in your test but also in the environment in which it is going to happen.
“If you torture data long enough, it will confess to anything you like.”
This great quote from R.H. Coase is embedded in the signature of one of the data analysts I work with. Indeed, people draw all sorts of conclusions, usually ones they prefer, from the results my team presents. One of my pet peeves is when someone says that the results show a “directional” result. That is, one without statistical significance. I always cringe when one of these directional results somehow makes its way up to a member of senior leadership. I am careful to point out our lack of confidence in it ensuring that no one places any big bets on that result. Ensure that you have a sound result, without bias, with significance, before you go to your boss asking for an investment.
“Seek success, but prepare for vegetables.” – Quote from Inspirobot.me
I’m not a blind follower of AI. While we do use it and it does help with some of our processes, I don’t think it will give us the perfect answer, at least not any time soon. There is definite benefit to the automation of complex processes that learn over time to develop ever-better insights into what marketing effort will perform better. But I think it is too tempting to think this will quickly and easily solve all our problems; even with billions of records of data and dozens of industrial-grade data models running on stacks of severs ensconced with incredible code to enlighten our execution. The answer is still elusive. Knowing what works and why it works is hard to find. AI will help us get there, but it is a tool in the tool chest.
It takes a minute to learn and a lifetime to master
The best starting hand in Texas Hold’em is a pair of aces. Of course, this can change quickly as the game progresses. The worst starting hand can become the best ending hand. To figure out if you have a good hand or not: start with an idea, test it (many times), and see how it does. Have a well-defined hypothesis, make sure you have identified the variables, stick to your plan, operate in a controlled environment, and make sure you read the results correctly. Computers can help you out, but unlike chess, there is no Alpha Zero that can tell you the right answer all the time when it comes to marketing; you’ll have to find that yourself.