Beyond - a digital engagement consultancy - london, new york, san francisco

I love debates about measurement best practices, so I was thrilled when I saw that the Institute for Public Relations published a paper by Angela Jeffrey and colleagues supporting the use of “weighted media costs” as a replacement for Advertising Value Equivalence (AVE). The paper has already stirred a bit of controversy (also see comments here). But, I think there is some value in the paper that is getting lost in the debate—the more metrics you use to estimate PR success, the better you can use those to predict business outcomes.

First, I should provide some background into what will seem like an esoteric debate for non-research audiences. The debate centers around whether or not PR measurement should adopt or prohibit using media cost estimates from advertising rate cards to measure earned media. For those unfamiliar with the “weighted media costs”/advertising equivalency value debate, here’s a very oversimplified summary:

AVE Advocates: “AVEs are great because they can be shown in units of dollars which CEOs and CMOs understand.”

AVE Opponents: “PR is not the equivalent of advertising. There is no evidence that a half page of earned media has equivalent impact on business outcomes as a half page of advertising.”

AVE Advocates: “But AVEs have stronger correlations with business outcomes than clip counts or impressions.”

AVE Advo-ponents (the middle grounders): “Okay, there’s something to this whole AVE thing, but the name is a little misleading since it suggests that advertising and earned media have equivalent value. Since advertising rates aren’t really indicators of value but are rather about cost, let’s call the metric something like ‘media cost equivalency’.”

Jeffrey and colleagues’ paper is written in support of this AVE advo-ponent view. It essentially offers a new name for AVEs- “media cost” , provides a method for weighing the costs by a few factors including sentiment, and finally provides some studies that are intended to support the predictive validity of weighted media costs. The case studies assess the correlational strength of sentiment-weighted clip counts, impressions, and weighted media costs/AVEs with business outcomes. In most of the case studies, Jeffrey and colleagues find that weighted media costs have the strongest correlation with business outcomes (e.g., revenue), followed by impressions and then sentiment-weighted clip counts. In the end proclaim that weighted media costs are the “best” PR metric and announce a new “paradigm shift” in earned media measurement.

I generally agree that composite metrics, such as “weighted media costs” are going to be better predictors of business outcomes than sentiment, clip count,  impressions or media costs alone. But I’m concerned that some audiences will interpret this paper as showing that media costs are a better business outcome predictor than clip counts or impressions. If you look at the data really closely (read: “skip this paragraph unless you’re prepared for some serious statistics jargon”), there is no evidence that media costs are better at predicting business outcomes than any other metric (to be completely fair, the authors never explicitly state that it is). One issue with the results is that the authors use Pearson correlations to assess the relationship between variables over time. Using Pearson correlation coefficients on time-series data can be a huge statistical “no-no” (proper time-series correlations have to account for a phenomenon called autocorrelation, which even time-lagged Pearson correlation coefficients cannot do). This means that the r and R² values presented in the paper are inflated and that the actual correlations for clip count, impressions, and weighted media costs are probably much closer to each other than the Pearson correlations reported in the paper. Secondly, the authors compare 3 correlation coefficients for three tightly interrelated variables. Clearly, clip count, impression and media costs are going to have strong positive correlations with each other (as clip count goes up, so will impressions and media costs), and sentiment is being weighted in each of the three media metrics. Because the authors used Pearson correlation coefficients (which compare each of the metrics in a silo), we don’t know the unique contributions of sentiment, clip count, impressions, and media costs to business outcomes. A more rigorous analysis where each of these variables were included in a single regression model could reveal a completely different result. It’s possible, for example, that clip count is the strongest predictor of business outcomes, followed by sentiment, impressions, and media cost. But given the way the statistics were conducted, it’s not possible to make these sorts of comparative judgments about the predictive strength of media metrics.

Regardless of what can and cannot be gleaned from the case studies, the authors make a good point that “weighted media costs” are likely to be a very good predictor of business outcomes because they combine so many different metrics, including clip count, sentiment, audience size (impressions), as well as the “prominence” of the brand mention (e.g., how often it appeared in the story) and the credibility of the source. It’s a bit of a no-brainer that the more metrics you use to predict something, the better the prediction will be. A classic example is college GPA. College admission departments know that high school GPA, SAT scores, and the number of extracurricular activities listed on the application will be a better predictor of college success than SAT scores alone. The same logic applies here: clip counts combined with coverage sentiment, audience size, brand prominence, and publication credibility is going to be a better predictor than clip counts alone.

I believe that this last point is being overlooked in the AVE debate. A lot of people are hung up on the phrase “advertising equivalency” and how this metric can be misused and misinterpreted when presented in currency values. One of the key themes in all of these case studies is that measurement that combines many different metrics is likely to be better than just using one metric, let that be sentiment, impressions, media prominence, clip count, or a media cost-like metric. Each of these metrics say something unique about a brands reputation in the media and each metric probably deserves some attention in PR measurement reports.

I want to make one final remark on this paper and the AVE debate in general. There’s been concern that using weighted media costs as a measurement standard will be problematic since access to media cost databases are expensive and only a handful of vendors offer weighted media cost-like metrics (see Katie Paine’s comment here). Fortunately, as the authors admit, the media cost is really just comprised of the prominence of the news coverage and the credibility of the publication. This probably means you don’t need to have access to proprietary media cost databases to get the predictive strength of “weighted media costs”. It’s easy to calculate the prominence of a brand within an article (was the brand mentioned in the headline, lead paragraph, etc.), and credibility can be estimated using a variety of free metrics (Google PageRank is a great one for estimating news site credibility, for example). There plenty of creative ways that communications professionals can create weighted metrics that predict business outcomes just as well as “weighted media costs” without relying on VMS or similar vendors.