There’s No Such Thing as a Guardrail

A common pattern in product experimentation is:

1. A Primary Metric: The thing you want to move up. DAU. Timespent. Monetization. Whatever.

2. A Guardrail Metric: Something you want to make sure doesn’t decline. Latency. Satisfaction. Also monetization. Whatever.

So now consider this hypothetical: You attempt 100 experiments in a half. You only ever ship experiments that (1) move the primary metric up, (2) don’t move the guardrail down. At the end of the half, what happened?

1. Primary went up, guardrail didn’t move strongly.

2. Primary went up, guardrail dropped.

3. Other, unknown.

If your answer was “2” then please stop reading and go run some more experiments. If you answered “1” and are now asking “why?” then read on!*

Because the truth is: A guardrail metric doesn’t stop regressions, it only slows them. And depending on how much you care about regressions, you should be thoughtful how you treat them in an experiment.

The Reason

So why does the guardrail drop? Because in reality, you don’t know which experiments actually drop it, you can only guess based on experimental data. Over enough iterations, you’ll start to miss more and more. And actual regressions will pile up.

Let’s look at an example. Here’s a simulation of 100 experiments, with primary effect on the x-axis, and guardrail on the y-axis. The gray dots represent all experiments, the orange dots are ones we’ve accepted according to the criteria above. You can see quite a few experiments have actual negative effects (below zero on the y-axis) on the guardrail, and the avg. impact is also indeed negative.

Run your own simulations here (thank you Fable): https://claude.ai/code/artifact/9da168f6-3f2b-4346-8fb0-d0da3b10fff4

Note that this is true in a surprising number of arrangements. It’s true no matter the variance, the mean, the statistical power of the guardrail. There’s only one factor that matters: Correlation. This is only true if the guardrail is negatively correlated with your primary measure.

And this is a hugely significant fact. If you’re testing a new feature’s retention against latency - it’s usually true that new features make an app slower - they are usually negatively correlated. So all of this caution applies. But engagement with monetization? Perhaps not, they may usually move in tandem!

The Options

However, it’s also true that using guardrails does decrease impact on their metric. Again, using a simulation, you can observe the differing impact of different decision policies. Without any guardrail at all, caring only about the “Primary significance,” you get both more primary movement and more guardrail regression than the usual “Naive guardrail veto.”

There are also other options!

1. Ratio or Compound Metric: You can change your primary metric to include the guardrail. Instead of “Posts per user” you can measure “Posts per User - 0.1 crashes per user.” This is mentioned as something Amazon does in Trustworthy Online Controlled Experiments.

2. Non-Inferiority Test: You can change the regression test. Instead of saying “the 95% confidence interval must contain or be above zero”, you can say “the 95% confidence interval must be above zero [or some other number].” Spotify appears to implement this.

3. Point Estimate ≥0: You can simply assert that the estimated regression, regardless of the confidence interval, must be greater than zero.

4. Many More.

Which one works best? There’s no free lunch here, there’s a consistent trade-off across the board, and a clear Pareto frontier. With only one possible exception…

Maybe You’re Actually a Bayesian?

There’s one approach that can possibly yield a win-win: Using empirical Bayesian priors. In the case where:

1. You have significant data on past experimental trade-offs between metrics.

2. That form a somewhat reliable prior on their future trade-offs.

You can theoretically do better than all the other approaches. I have never worked at a company where I’ve seen this work, so take this with a big old “kids don’t try this at home” warning.

So what should you do?

1. Don’t use guardrails and think they actually stop regressions.

2. Do pick experimentation decision criteria according to the trade-off you want to make between metrics.

3. Let’s all work together to think of a better metaphor than “Guardrail.”

*If you answered “it depends on the correlation between the Guardrail and the Primary, then you should be writing this instead.

Code guided with AI assistance. Words, as always, by a human only.