Skip to content Skip to navigation

On the Near-Impossibility of Measuring the Returns to Advertising - Randall Lewis

young man with styled brown hair in front of brick wall
April 15, 2013 - 12:45pm
UC Berkeley in Room 330 in Blum Hall

Randall Lewis, Senior Economic Research Scientist in the Knowledge group at Google


Twenty-five display advertising field experiments run at Yahoo!, amounting to over $2.8M worth of impressions, give insight into the volume of data needed to form reliable conclusions concerning advertising effectiveness. Relatively speaking, individual-level sales are typically volatile, and only \small" impacts from advertising are required for a positive ROI. Using data from major U.S. retailers, we present a statistical argument to show the required sample size for a randomized experiment to generate sufficiently informative confidence intervals for a given campaign is typically millions of individual users exposed to hundreds of thousands of dollars of advertising. The argument also shows that sources of heterogeneity bias unaccounted for by observational methods only need to explain a tiny fraction of the variation in sales to severely bias estimates. Measuring advertising effectiveness is thus a situation with low-powered experiments and faulty observational methods | precisely where we would expect poorly calibrated beliefs in the market

The talk is based on joint work with Justin M. Rao

Event Sponsor: 
the Institute for Research in the Social Sciences (IRiSS) and the Graduate School of Business (GSB)
Contact Email:
Contact Phone: 
(650) 724-5221

This event belongs to the following series