A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments

By Pavel Dmitriev, Somit Gupta, Garnet Vaz, Dong Woo Kim | February 24, 2019

I saw this presentation at KDD and thought it provided hard-won insights into the types of interpretation problems one encounters when performing online tests. I’ve run into some of these issues myself, but this paper is still the best overview I’ve seen of problems you might not think about until they bite you.