Research Papers

A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments

I saw this presentation at KDD and thought it provided hard-won insights into the types of interpretation problems one encounters when performing online tests. I’ve run into some of these issues myself, but this paper is still the best overview I’ve seen of problems you might not think about until they bite you.

Continue reading