Cliff Moon's session from Monitorama PDX 2023. The traditional statistical models used in A/B testing are built to support product decision making around things like buttons clicked, messages sent, etc. In other words normally distributed metrics. However what happens when we want to make decisions about the performance impact of an experiment? Performance metrics are decidedly non-normal, and typically subject to a long tail. Averages of such a dataset only have enough precision to surface the most egregious performance degradations. In this talk we'll discuss the development of a system for catching performance degradation of A/B experiments in an environment with thousands of concurrent experiments at any give period of time.