Eytan Bakshy - Adaptive field experiments using Bayesian optimization

Abstract: Online experiments ("A/B tests") are the workhorse of modern Internet development, yet these experiments are generally limited to evaluating the effects of only one or two variants.  In many cases, however, we are interested in evaluating the effects of thousands or a potentially infinite number of possible interventions, such as treatments parametrized by continuous variables, or dynamic contextual policies that map particular states to different actions.  I will discuss a new approach to large-scale field experimentation using Gaussian process regression models and Bayesian optimization.  Using empirical examples, I will show how we are able to effectively make predictions about yet-to-be-observed treatments, and make substantial improvements to applications ranging from optimizing mobile software for emerging markets to improving machine learning systems.