From “Goat” to “Hero”: The Saga of the IAB Summit

The above quote was included in a statement that Randy made in the press release about the summit meeting we attended with the IAB last week. Comscore met with the IAB to discuss the issues related to measuring the size of online audiences. We did indeed commit to an MRC audit and are currently in the process of completing the pre-audit. From the IAB’s press release, it is easy to see that the meeting was not simply about accepting MRC audits. A lively discussion that covered a wide range of issues resulted in new IAB commitments that we believe will solve many of the problems cited in its April 2007 open letter.

Comscore has invested millions of dollars in research and development in the building of our measurement capabilities. Our management team includes leading market researchers with rigorous research and scientific training. In fact, we’ve created many of the innovations that are fixtures in Internet measurement today, while maintaining strict standards for data quality. Nevertheless, there is some confusion in the marketplace about what we (and Netratings) do and how we do it. We have naturally been protective of our propriety methods, because many times we’ve seen competitors rush to copy a methodology or a product that we pioneered. However, we are willing to disclose what we do to certified auditors under strict confidentiality protections, so that they can verify that we deliver what we say we do.

One of the chief elements in the controversy is the discrepancy between the estimates of Web site unique visitors using a panel approach compared to the approach used by the publishers of these Web sites. Many of the publishers’ numbers, derived from their server logs, indicate audience levels much higher than what our data show. Because of this, a few months ago we initiated a research project to examine why there are systematic differences. We believe we have found the main causes.

With the server-log approach, cookies are placed on visitors’ computers to identify “unique visitors,” so that repeat visitors are not counted more than once. This approach only works, however, until the user clears the cookies from their PC -- either in the course of normal PC hygiene or through the use of security scanning software (a common practice today). It turns out that the frequency of clearing cookies is much higher than many might previously have believed. About 30% of Internet users clear their cookies at least once in a month, but some users clear them at every session and can be counted as unique visitors by server logs more than 10 times in a month. This multiple counting dramatically inflates unique browser estimates derived from web logs -- which are, essentially, counts of unique cookies. Furthermore, about 12% to 15% of machines block cookies completely. A publisher who follows the methods described in the appendix of the “IAB Ad Impression and Audience Measurement Guidelines“ (2004) would identify such machines with a unique combination of an IP address and a user agent string. But, here again, the result is tremendous inflation as our research shows the average home PC has 10.3 IP addresses in a month. This group is similar to the group deleting cookies, only with more inflated audience counts.

Finally, we find that some publishers fail to remove international traffic from their domestic web logs. When they compare their worldwide traffic to our reported U.S. estimates, they naturally find differences. When each of the above factors are present, the overstatement by web logs can be as high as 1000%.

To summarize, we can explain most differences between our numbers and web logs with the following simple equation:

Difference ≈ Unfiltered international traffic + Inflation due to cookie deletion + Inflation due to cookie blocking

The accuracy of Comscore’s data has been confirmed many times. For example, when we examine measures that are easy to define – such as how many dollars were spent on a site or how many advertising impressions are served – we are typically close to what site operators report. Our online sales projections are also within a few percent of what the U.S Department of Commerce reports in their quarterly e-commerce sales estimates.

Overall, we believe that the IAB meeting had a positive outcome. We communicated our views on the causes of web log inflation and pointed out that the IAB’s own guideline appendix would lead to inflated numbers and should therefore be revised. We also made the case about the critical role that panel data plays in measuring the interactive industry and the ROI from online marketing. As usual, when reasonable people sit down together, mutual understanding can occur. The IAB now recognizes the value of panel data:

“The IAB acknowledges that there is a role for both panel-based and server-based metrics in the measurement of audience and both will continue to be used by agencies and advertisers as a combined decision-making tool.”

The IAB has also committed to revising outdated standards for server-based measurement and has suspended audits based on existing standards. The IAB is also committed to auditing all measurement techniques, including web logs. We believe these audits will eliminate the use of inflated server log data – affected by cookie deletion rates, international traffic, spiders and bots, and other factors – and will help reduce the magnitude of the measurement gap, thereby reducing the measurement controversy that currently undermines the credibility of the industry. In addition, the IAB has committed to promoting efforts to educate the industry about the differences between panel and web log estimates, and the likely causes of those differences.

“The IAB agreed to intensify its efforts to arrive at a definition of unique visitors, page views and time spent. This process will include reviewing the impact of cookie deletion, international traffic, spiders and bots and other potential factors.”

"... The IAB agreed, in response to requests from the audience measurement companies, that it will recommend suspending further audits against the initial audience-metrics guidance in the appendix to the 2004 Ad-Impression Guidelines. Auditing should take place once industry accepted guidelines have been created.”

Although this meeting started out as a challenge to our methods, we hope that it may prove to be a turning point for the industry. By gathering together representatives from publishers, advertisers, and measurement companies – using both server-log data and panel-based data – we participated in an important moment to confront the issues of different measurement techniques. We look forward to moving beyond the stage of debating measurement techniques and toward using all of the data to its fullest potential.

To bring clarity to these issues, in the coming months I will post a series of “Web Measurement 101” briefs, which will further address topics such as disparities in Web logs data versus panel-based measurement.

If you would like a copy of Comscore’s white paper study of cookie deletion, please visit www.comscore.com/request/cookie_deletion.asp and we will forward it to you.