Will measuring digital signage ads really do us any good?

Author: Bill Gerba on 2008-01-28 11:20:05

Over the past several months, WireSpring has been evaluating various strategies for "measuring" digital signage. I'm putting this term in quotes because, like virtually everyone else, we still aren't sure what we're trying to measure, or how to best go about it. With our clients asking about the available options, we hoped to see if we could settle the issue once and for all. Unfortunately, after talking with numerous companies and evaluating many of the options available today, I feel like we're not much further along than when we first began our research. But we did uncover something interesting: Far more important than the actual method of measurement, or even the decision of what should be measured, is the understanding of the value of the data that will be collected. In other words, even if we can get accurate metrics, are these really worth anything to ad buyers and other decision makers?

Not sure what I mean about the "value of the data?" Well, I'll put it this way: If I were to randomly ask 1,000 readers of this article why measuring the reach or impact of digital signage is important, I'd probably get ten or maybe twenty answers. (You know the common ones: it makes ad space more valuable because people enjoy and act upon the ads, it helps in ROI calculations, or it contributes to the other measurement goals we discussed at last year's Digital Signage Expo.) Now, what if I were to ask the same 1,000 people how much that was worth to them? Chances are I'd get close to 1,000 different answers. And that, folks, is the number one problem that must be solved before getting serious about measurement. There are lots of different things that we can measure, and lots of ways to measure them. But when push comes to shove, the value of the measurement is the critical factor in making the business decision to expend resources on the process.

Let me give you an example: a few months ago, we were working with a client who manages an advertising-driven network in about 50 retail stores. The client paid for about 60% of the upfront installation costs, and is responsible for 100% of the maintenance costs. They're also responsible for 100% of advertising sales, although the retailer helps by providing access to their vendors. (There's no co-op marketing program at this retailer, however.) For several years, they've been selling ad space at about a 30% subscription rate, which has allowed them to be profitable and grow organically (albeit at a fairly slow pace) into other adjacent vertical markets. In search of ways to either sell more space on their network or else make more money on existing clients, they started looking into measurement options. Right from the beginning, this proved to be more difficult than expected. For example, as I pointed out, how would measurement allow them to make more money on existing customers? Those customers -- most of whom had been advertising for months or even years on the network -- had already demonstrated that they valued having their ads show up in-store. Whether or not they had formally determined the effectiveness of the medium (and I'm guessing that some did and others didn't), their continued participation meant they were satisfied with its overall performance, regardless of how they "measured" it. Thus, it seemed unlikely they'd find any newfound value in getting monthly data with some nebulous metric like impressions, opportunities to see (OTS) or "engagement factor," and even more unlikely that they'd be willing to pay much for the privilege.

As for signing up new customers, our client was unsure of whether they had missed out on any opportunities because of their lack of formal measurements. While coming up with a good metric and demonstrating the ability to produce results would certainly improve their sales pitch, it would only make a big difference to those advertisers who valued their particular metric as much as they did. So, for example, if the network company did decide to measure OTS, but their advertisers had no notion of what an "opportunity to see" was worth to them, it would give them little additional incentive to participate on the network. This isn't to say that research programs like Nielsen's PRISM and the ongoing work by POPAI don't hold significant value for our industry. Rather, the point is that network owners may have to do some legwork to help advertisers translate these newfound metrics into terms they know and understand.

Before you jump down my throat screaming "just measure sales lift already!" there are two things I'd like to point out. First, there are lots of ways to gauge sales lift, and second, believe it or not, sometimes that's not the right thing to measure. I'll start with the second point, again through an anecdote. Just the other day I was speaking with another client who had just started testing a digital signage campaign to advertise a brand new Unilever product. Unilever, who is clearly not a novice when it comes to product launches, indicated that for the first month or two after the product launch, the only thing they wanted our client to measure was product recall. From decades of experience they learned that creating buzz and being memorable was more important for establishing the product as a success than early sales numbers. So in this case, the advertiser knew what they wanted to measure and they obviously found that measurement to be valuable. But because of the nature of the measurement, our client used a series of decidedly low-tech methods to gather their data, since this particular metric doesn't lend itself well to technology-driven solutions. (For a few examples of the methods that work, see my article on retail media tracking from 2006.)

But what about those cases where we do want to measure sales lift? What's the best way to do it? There are numerous options ranging from hand counting to register receipts to path and gaze tracking -- but what works best? After months of studying the problem, I'm still convinced that if you want to look at sales trends, the only way to do it is studying the sales themselves. That means setting up controlled, A-B split tests and measuring results using real data from sales receipts. I know this answer is going to upset a lot of people who have been hoping that some new and wonderful technology will come along to make the entire process of measurement easier and more automated. Unfortunately, every other alternative we've looked at has either introduced uncertainties that spoil the data, or relies on an alternate measurement as a proxy for the real thing.

Still, this is one area of the industry that's changing quickly, and I'll be the first to admit that we've probably excluded a few measurement techniques. That's where you come in:

Have you had a good (or bad) experience trying to measure the effectiveness of your digital signs? What about your static POP displays, posters, and other out-of-home ads?

I'd love to hear your first hand reports -- leave your thoughts and comments below!

Comments (17)

Subscribe to comments for this article | Trackback

2008-01-28Tim Goltz writes:

Why don't network providers (who ostensibly hold much of the power to measure) offer a "menu" of typical measurable items? In other words, have a customizable list(s) of what the agency/advertiser might like to measure and let digital media networks finally move towards the user-driven type of media they keep touting? Whatever the choice(s) of what measurements are to be made, reasonable costs can/should be associated with such choices.

If, as in the article, clients such as Unilever "know" that the most important metric on a given product rollout is "recall", they should be willing to pay well for that type of option associated with those network(s) on which they are rolling their product out. Of course, this "menu" of services must have sensible limits of what types of measurements are offered...

Why keep on searching for a holy grail (i.e., the "right" kind of digital signage metric) which will likely end up being (at least somewhat) different for each client?

2008-01-28Bill Gerba writes:

Tim,

I think the fundamental concept of what you propose is valid, but in my experience (working with a number of networks), I don't think a standardized a'la carte menu will work on any reasonable scale for two reasons:

First, it's the advertiser, not the network provider, that indicates which metric is important. In other words, just because a network provider can measure the number of engaged views (or whatever), that doesn't mean the advertiser will find it valuable.

Second, companies like Unilever "know" simply because they've been working at it for a long time -- decades, in their case. But that doesn't mean that they necessarily have a complete understanding of every new product rollout, or that there isn't something new for even them to learn.

I agree that there is no holy grail, and we certainly shouldn't be searching for one. But the point of today's post is that all too often companies get stuck on the idea of measuring something -- anything -- without first figuring out what they want to measure, why they want to measure it, and most importantly, how much that measurement is worth to them.

2008-01-28Jill Ruttenbeg writes:

I really appreciate your blogs and the valuable information they provide. Our company has approx. 275 locations that we maintain and sell advertising for. You are correct, and like you, we are trying to come up with the magic tool to measure effectiveness. We find if we try to translate the advertisements directly in to increased sales for the advertiser - it will be difficult. We focus instead on impressions per minute and "branding". Now if you could only provide me with good sales people!!!!
Thank you.

2008-01-29Fran�ois Reeves writes:

I agree with Jill. Bottom line, sometimes a sign is just a sign (albeit it is multimedia) and we are selling impressions. The multimedia dimension adds value, it is eye catching, superior to flat art and can be enriched with sound and updated remotely. All monitors should be equipped with a motion detector capable of recording a fairly good representation of traffic. We have just begun tapping on the media's potential. It is stronger when it is networked, it is stronger when it is tactile (customer surveys anyone?) it is even stronger when it is positioned in terms of demographics. I'm getting sidetracked sorry. My point is that yes we can use traditional metrics but we have to find a way of value adding specific media qualities to a given campaign. This media, as anyone in front of a demo will see, is superior to print in delivering eye catching, longer lasting and adaptable impressions.

2008-01-29Akis Liantzouras writes:

Hi guys. We have a previous experience on classic media and POP. I mean that metrics of TV,radio etc for decades are not really counting effectiness but potential reaches and the target was awarness. Now how exactly that is mirrored in sales lift is a magic spell all advertisers keep as a secret ( since nobody ever gave a good explanation). Or outdoor...advertisers count the traffic data to help themselves decide where to put their money (this is really a joke if you ask me).
Therefor I want to say that our medium is really effective and we should give advertisers to understand that is an all new thing and should be measured like that. I believe personally that a new metric should be invented .

2008-01-29Bill Gerba writes:

Jill: thanks for bringing some additional perspective from the field. However, "impressions per minute" seems a little confusing to me. I can understand selling "minutes of content" since there's a limited supply of screen time and I can understand selling "number of impressions" since that's the most basic (meaningful) number to measure, but what's the added benefit of doing both at once? More importantly, what does that metric actually indicate to the advertiser? (maybe I'm just being dense).

Fran�ois: I agree that, "the multimedia dimension adds value, it is eye catching, superior to flat art and can be enriched with sound and updated remotely", and those are certainly the selling points that most network providers are pushing these days -- they're the most tangible, obvious, and available. Measurement, though, is becoming more important, and while it isn't going to make any of those other points less important, we need to have a good argument for what measurements to use, and how to value them. You seem to favor an impression-based metric as the most valuable -- and that may be correct -- but that's still a long way off from being able to say how much the ability to measure is actually worth, and (separately) what each impression is worth.

Akis: Any suggestion for what the new metric should be? :)

2008-01-29Tim Goltz writes:

Bill:

You are, of course, correct that the advertiser holds its own "key" metric(s). More basically, what I tried to suggest above is that the whole process advertiser-agency-network move towards more conscious collaboration. This is where (as you put it) the "a la carte" menu may become useful - it would suggest at the outset of the advertiser-agency-network relationship that the advertiser should also be thinking about various ways it might ask questions as the "menu" evolves over time...

2008-01-29Tony writes:

There are big differences between the academic reasons for using in-store media or digital signage and the practical reasons.

Twelve years ago, if your competitors were on the Internet, you had to be on the Internet. I'm sure that 100 years ago if your competitors were selling via telephone, you had to sell by telephone.

The same is true today. If your competitors are out there catching additional eyeballs, you'd better get in the game, or at least understand the game, so you can enter when the academic reasons become clear.

2008-01-29Bill Gerba writes:

Tim: More collaboration between the parties would definitely help out a lot! And who knows, maybe NBC's recent digital signage upfront is a sign that that's finally coming about. But I think we're still far away from having an accepted menu of measurements and related services that all networks can agree to implement, and all advertisers will agree are useful/valuable :)

Tony: There's no question (in my mind, at least) that marketing at retail is valuable. But what do you think is the added value of measurement, and/or what are the relative values of each kind of measurement and each method of measurement? Those are the big questions now.

2008-01-29Fran�ois Reeves writes:

Of course we are all assuming that "traditional" measurement is exact and that advertisers get their money's worth. This is inexact but agencies, clients and broadcasters all abide by the same white lie for lack of a better method of evaluation.

The Internet is much more precise and agencies and clients have heavily started to move money to these new metrics. Digital signage will explore various ways of measuring until it finds a consensus between the media, the clients and the networks. The industry has to find its white lie.

2008-01-29Jason Goldberg writes:

I don't have much experience with success criteria for advertising based networks. But I imagine the principal is the same as measuring efficacy for POP/Visual Merchandising projects.

1. Agree with the stakeholders on the success criteria. We have lots of retail clients that define it as lift in sales, but they have very different ideas about what methodology is most reliable. Let's say it's a matched panel test (test and control stores, normalized based on historical year over year and leading months sales data).

2. Run the test to determine a sales lift and correlate that sales lift to store traffic, or department traffic, or interactions, or whatever metric you have for your display impressions. Now you have a ratio of impressions to incremental sales dollars.

3. Test your average sales lift vs store traffic, for all the panels against the same ratio for each store (to determine the standard deviation).

4. If you have an acceptably low deviation, then you can now use the ratio to extrapolate the likely sales lift in any store where you deploy the experience. If the deviation is high, then you need to increase your sample size.

5. Retest periodically to keep your ratio's accurate.

I do a case study at some of the industry events where we walk through a very detailed matched panel evaluation for a branded interactive Levi's display inside of Sears and JC Penny stores. It used 5 panels of 6 store each, to predict sales lift in over 600 stores.

2008-02-02Bill Gerba writes:

Hi Fran�ois: The industry has to find its white lie. - I love that line. Cynical, maybe a bit sarcastic, but so true! I agree, there's some kind of implicit agreement that all parties have to enter into. The terms are still up in the air.

Jason: Thanks for that great break down. We've had customers take very similar approaches for digital media, so I'd say that the core methodology is interchangeable for static POP and digital out-of-home. When it comes down to it, it's hard to beat a well-controlled split test experiment, especially when you already have the means to translate results from a small sample into predictions for a larger group.

2008-02-05Axel writes:

If you get good sales people that understand how the medium works, they can sell it without delivering tons of hard data to the advertiser and in most cases the advertiser will get their own ways to measure it without us having to give them that info (focus groups, shopping trips, sales behavior, etc) because better than anyone, the advertiser knows what and how they want to measure a medium and it varies from advertiser to advertiser. Like someone here said, not everyone measures the same thing.

I'm not saying that Hard Data is worthless, I'm just saying that this is a new medium still in diapers having a hard time to grow and that I don't think that the use of the most advanced hardware and software to measure it will make advertisers turn to Digital Signage as we all want.

Even if you get lots of numbers and hard data obtained from the most advanced gear on measuring audiences you will always need a good sales person that can use that information to sell a new medium like this.

I think it depends more on how well a sales person does its job rather than how many money you invest in technology to convince anyone that the medium works.

I also think that it's more important than anything to train the advertisers on how to use, buy and measure Digital Signage because the concept is so broad that sometimes they don't buy it because they don't understand it and they go for the safest way... they buy traditional media.

2008-02-07Neil Steiner writes:

Measure exposure on a digital sign? Is that important?

Bill Gerba has written in the past about differences on CPM cost of traditional vs. digital media. So it would seem that this is important if you are an agency buyer. But maybe not if you are hawking space sales on a narrowcast network in bars in doctors offices. For thoses, let's not make it too accountable.

But what about direct marketing feedback? I've noticed Wal-Mart doing this in the check out line as you swipe your credit card and sign the small touch screen LCD display. They can ask did you like questions or will you questions or how was your shopping experience questions.

What if we did this on a larger LCD display at POD(Point of Decision) on iDS (interactive Digital Signage). Would this have value to CPG marketers or media agencies? How would this relate to ROI? Would this change the media from a pay for exposure model to pay by response one?

I would also like to know if anyone else is exploring this or has it been deployed in a digital signage or kiosk platform?

2008-02-08Bill Gerba writes:

Hey Axel: Do you think that's likely to remain the case, or will advertisers become more demanding as other media continue to refine their own built-in measurement capabilities? Will P&G, for example, continue paying to advertise in-store once TV-on-demand can deliver their ads to a super-targeted audience and give them immediate and accurate feedback? What about mobile and Internet advertising, with their ability to establish one-on-one connections? (I don't have an answer, just playing devil's advocate :)

Neil: "it would seem that this is important if you are an agency buyer. But maybe not if you are hawking space sales on a narrowcast network in bars in doctors offices. For thoses, let's not make it too accountable." I definitely get the feeling that this has been the unofficial motto of the industry, and for purposes of growth and establishing the medium, I don't disagree with it (hell, I'm a free market capitalist -- I say if you can sell it, it must be worth something to the buyer, right?) As the small guys (a) try to grow, (b) get bought or merged, and/or (c) try to sign up new and bigger advertisers, I expect there to be more emphasis put on measurable results, though.

2008-02-10Axel writes:

I think it will remain to be the case.

When you talk about measuring TV-on-demand, this concept uses more or less the same measuring way that broadcast TV does, so it's easy to measure... rating. (and rating is soooo subjective but nobody questions it)

When you talk about Out-of-Home you can go from a screen in a toilet to a Las Vegas strip digital billboard so I don't see how anyone will be able to develop a way to measure it in general when every product depending on where you are advertising it will end up needing it's own measuring system.

Trying to make myself clear here, let's take your example with P&G. If they happen to get in their marketing plan for 2009 "Digital Signage" as another row in their excel spreadsheet next to advertising on TV, Billboards, Magazine, etc. What would they want to measure to prove that their investment was worthwhile????

On TV they get ratings, on magazines they get circulation, on billboards they get traffic/impacts and in Digital Signage what do they get?? better yet... what do we offer when we sell the medium???

So first you need to establish Digital Signage's placement. Is it going to be on a Store??.. ok, then what does P&G wants to measure there??? Visitors, tickets issued, people passing by the screens, units sold... what??? And what do they want to obtain from advertising in the store... brand recall, sales lifts??? what???

And also what kind of advertisement spot will you create??? Is it for a promotion, for a traditional product with only brand presence, or is for the introduction of a new product... another big "what" in the equation.

There are simply so many variables that I don't think a single measuring system can deliver the Hard Data the advertiser will want to see or the Hard Data that will pour budgets on our networks.

That's why the SALES PERSON it's the most important link in the chain. He or she has to convince the advertiser that the network works for more reasons that only for it's Hard Data result

2008-02-21tim Hori writes:

Axel: I agree with you. At the end of the day it all boils down to the ability of the sales person. You can have all types of data, and if the decision maker for signing the placement budget isn't emotionally sold on the idea, you're going to have a hard time getting the money.

Because I've experienced talking to and completely convincing advertisers to place in our venues with NO data or measurments. I simply was able to paint a strong and clear enough picture in their heads that they understood the value of placing In-store.

What we offer is the future of advertising in many ways (at least i believe it is). But it is still new, and I find that most advertisers or agencies are still very clueless to how to effectively use In-store advertising. But of course, they would never want to admit that. So, instead of saying "Hey I have no idea how to effectively use this new media, can you help me?", they just place their money on what they know and feel like an expert on, ie. Trad media. And I don't blame them. So it goes back to the sales person to feed them with the ideas, the vision, and the knowledge needed for them to present In-store advertising to their bosses without looking like a fool, so the budgets get approved.

Show more articles on this topic

Digg this! |

Del.icio.us

Previous Article: An open metric for pricing digital signage ads: vJive's SCQ
Next Article: How much would you pay to measure digital out-of-home ads?

Front page of Digital Signage Insider Blog

The Digital Signage Insider
WireSpring's blog featuring tips and analysis from a team of industry experts