Calculating Digital Signage ROI: Understanding the Limits of Your Data

Author: Bill Gerba on 2005-03-30 12:02:13

This is a continuation of a previous article called Calculating Digital Signage ROI: The Ground Rules. In this article, we're going to focus on some things NOT to do while performing an analysis of your digital signage data.

Understand the Limits of Your Data Set
We've all heard the saying "numbers don't lie," and that's true. However, it's unfortunately also true that people do lie. Methodically. And perpetually. And in fact, even the best-intentioned people can get into trouble when analyzing data because their own hopes and beliefs will alter the way they interpret their numbers. So I'd like to point out a couple of common pitfalls and misunderstandings about data analysis that will hopefully spare you the embarrassment of being called a cheat and a liar the next time your customers ask you for the hard numbers.

First, correlation DOES NOT imply causation.
In other words, just because two events happen to occur at about the same time, you can't automatically assume that one of them caused the other -- or will continue to do so in the future. Without a control case, you'll never be able to accurately measure the impact of your digital signage network. You might like to think that the 30 second spots that you're running for cotton tube socks were responsible for the 15% up-tick in tube sock sales, but in reality, other factors -- perhaps unusually cold winter or a new fashion fad that idolizes 1970s basketball stars -- could have caused the surge as well. The only way to be sure is to measure your signage-enhanced numbers against numbers from a known control group, for example a store with similar demographics and geography, but no signage network. If I've confused you with my explanation of this concept, you might want to check out this link, which gives a brief introduction to some common statistical notions.

Next, work blind when you can.
As I said before, even the most honest people can be influenced by their own subconscious desires. That's why any serious scientific analysis takes place under "blind" or "double blind" conditions, where researchers are not aware of whether they're analyzing their target dataset or that of a control group. Digital signage analysts can work much the same way by simply removing any identifiers from their data set and having more than one person analyze the data. If you can afford it, you might want to consider having an independent auditor come in to check your numbers. The bigwigs like ACNielsen and Arbitron have all sorts of clever ways for measuring traffic in your locations, and years of experience in determining the effectiveness of marketing campaigns in retail locations. Also, paid auditors can lend an air of legitimacy to your data, and they have (slightly) less incentive to cook the books (Arthur Anderson consultants notwithstanding, of course).

And finally, don't affirm the consequent
When people say something like, "you can't prove a negative," what they're probably talking about is the 3,000 year-old proof that you cannot prove an argument true by affirming it's consequent. Consider the example:

[ 1 ] If the digital signage network is effective then we will see an increase in sales of promoted items.
[ 2 ] Sales of promoted items have increased.
[ 3 ] Therefore, the digital signage network is increasing sales.

These are perfectly logical-sounding statements, and a great way to "prove" the same kinds of things that I mention in the "correlation does not imply causation" example above. The problem is, you actually can't prove this argument. Even though the premises might all be true, the conclusion is not necessarily implicated by the first two clauses.

I think this is a good place to leave off for now. In a future article, I'll start looking at different ways to model ROI, as well as some techniques for getting the data you need to tweak your network's performance.