Nobody Knows What the R Stands For

I now “manage” a team at work. This involves a lot of exciting new changes in my life, mostly involving who now takes me more seriously, and who shouts at me when things go wrong.

Not that the shouting ever really happens — we’re all very friendly folk at PLACE-OF-WORK. But sometimes I’m caught by some humbling and humiliating noise.

When, for instance, Tim turns to me in his chair and says “did you know that the submissions app is down?”, reader, I did not know the submissions app was down. I was doing something else. And when a colleague from another team says “did you know that you haven’t been sending us drafts since 4 o’clock yesterday morning?”, mea maxima culpa, I was equally ignorant.

And humiliated. It’s my system — I am, sadly, responsible for what it does. So if my chief way of knowing that something is wrong with it is someone outside the team complaining (engineers, product owners, or — worst of all — actual users) then I feel pretty stupid.

Tim, I should note, is not on my team. When Tim knows before I do, something has gone badly wrong with how I find things out.

A Word About the Acronyms

I have spent an embarrassing amount of time reading about incident response metrics, and I cannot express to you how much I have hated most of it. There is so much noise. Everyone has a four-letter acronym. Everyone’s four-letter acronym means something slightly different from everyone else’s.

The worst offender is MTTR. Mean Time To Recovery. Or Repair. Or Remediation. Or Resolution. Or Restore. Nobody agrees, nobody checks, and two people can have an entire meeting about “improving our MTTR” while meaning completely different things.

So let’s not use it.

Thank you to the Google SRE book¹ for boiling this mess down to two numbers I actually understand and care about. Before we get to them, though, here’s the number we don’t care about.

Two Strategies

There’s a school of thought that says: slow down, make errors less likely, and you’ll have fewer problems. This is the waterfall instinct. Release management, code review gates, long QA cycles — all aimed at making failures rarer by putting more distance between them. In reliability terms, this is optimizing for Mean Time To Failure (MTTF): fewer releases, fewer opportunities to break things.

The problem is that this doesn’t actually eliminate errors. What it does is batch them. You pile all your changes up and release them in one big terrifying lump, and when something breaks — and it will — the break is bigger, and you’ve optimized your entire process for caution rather than speed. Discontinuous deployment, discontinuous pain. You can’t fix quickly what you’ve built to deploy slowly.

Agile and XP make a different bet. They say: fine, failure is constant. Accept it. It’s easier — and safer — to recover from a steady stream of small problems than to survive one catastrophic release. So instead of avoiding risky things, we make risky things less risky.

Which, oddly enough, is exactly what Google worked out too. Something like 70% of failures are caused by changes², and we cannot stop changing our systems. Software development is running with scissors. It is inherently sharp and dangerous, because of the key tenet of agile:

nobody knows what they’re doing

Product managers don’t know what we need to make, UX don’t know how our users work, engineers don’t know the right way to build it. We never know the total effect of our changes. So don’t optimize for making failures rare. Optimize for recovering from them fast.

And here’s the nice part: the practices that help you recover fast — TDD, continuous delivery, clean code, quick pipelines — are the same practices that make failures less likely anyway. You get MTTF for free by chasing the other numbers. You just don’t aim at it.

So which numbers do we aim at?

The Two Numbers I Care About

Recovery isn’t one thing, it’s two questions:

1. How quickly do you find out something is wrong?
2. How quickly do users stop hurting?

The first is Mean Time To Detection (MTTD). The second — from the failure to the users being okay again — is Mean Time To Mitigation (MTTM). Both are measured from the moment of failure. MTTD is how fast you notice. MTTM is how fast you stop the bleeding.

The gap between them is where your pipeline lives. All the detection in the world doesn’t help if, having spotted the problem in thirty seconds, you then spend forty minutes waiting for a rollback to deploy.

Two different problems. Two different sets of tools to fix them. Let’s take them in the order they happen.

Detection: Oh Look, We’re Bleeding

One question: how do you know what your system is doing?

If you have metrics but no alerts, you have a dashboard. Dashboards are lovely. Dashboards require someone to stare at them. Nobody stares at a dashboard all day, and if they do you should give them something better to do.

What you need is an alert that fires when no submissions have been received in the last five minutes. This is easy to configure in Grafana or whatever you’re using. The technology is not the hard bit. The hard bit is knowing what to alert on.

The guiding principle: alert on symptoms, not causes. “No submissions in five minutes” is a symptom. “Database CPU at 80%” is a cause. The symptom is what your users feel; the cause is what you go and investigate after you know there’s a problem. Alert on the thing that hurts, not on every thing that might one day cause hurt.

And then there’s the discipline problem. I’ve had the alerts. I’ve ignored the alerts. Why? Because the alerts channel was a firehose. Every noisy, badly-tuned rule pumping constant garbage into Teams meant that when a real alert arrived, it looked exactly like all the fake ones.

The fire alarm that cries wolf gets switched off.

So clean up your alert channel. Ruthlessly. An alert should be an exceptional, unambiguous, someone-needs-to-act-right-now event. If it isn’t that, it isn’t an alert. It’s noise wearing an alert’s clothes.

Mitigation: Just Stop the Bleeding

Now the failure is happening and you know about it. The question is not “what’s the bug?”. The question is: how do you stop the user pain, as fast as possible?

Here is the thing that took me an embarrassingly long time to internalize. Usually the answer is not “fix the bug.” Fixing the bug takes time you don’t have, and it takes it while users are actively suffering.

The answer is: roll back.

Ship a broken change, detect it, roll it back, and the users stop hurting. Then you fix the bug — properly, carefully, at a human pace, not white-knuckled at 4pm with everyone watching. The fix and the mitigation are different events. Confusing them is how you end up debugging in production while the site is on fire.

This is why fast pipelines matter, and it’s a reason people forget. A quick pipeline isn’t just for shipping features. It’s for shipping rollbacks. A pipeline that takes forty minutes is a forty-minute floor under your MTTM, every single time, no matter how fast you spotted the problem. Your ability to un-break things is capped by your slowest deploy.

Fast, confident pipelines aren’t a luxury. They’re load-bearing infrastructure for your MTTM. And the confidence comes from tests: a green suite on a clean build is what lets you roll forward with a fix and mean it, instead of nervously refreshing the dashboard for an hour hoping you got it right.

How to Actually Use Them

Two numbers, one procedure. When an incident is over, look at the numbers and follow your nose:

1. Look at the time to mitigation. Was it too long?
2. If it was, look at the time to detection.
3. If detection was slow, you have an alerting problem — go and fix your monitoring.
4. If detection was fast but mitigation was still slow, you have a pipeline problem — go and make your rollback faster.

That’s it. That’s the whole diagnostic. The two numbers don’t just tell you that you’re bad; they tell you which thing to go and fix. Mine, right now, is detection. When I find out we’re broken, we fix it fast — clean codebase, good logs, quick pipeline, mitigation is fine. I’m just finding out too late. Every time. From Tim.

Capture the Numbers

So how do I know if my detection is actually getting better?

I don’t. That’s the problem. I think it’s bad. I feel like it’s improving. I count how many times Tim has embarrassed me this month.

That’s not data. That’s vibes. And vibes don’t scale.

What you want is to record, for every incident, three timestamps:

1. When did it go wrong? (start)
2. When did you detect it? (gives you TTD)
3. When did you mitigate it? (gives you TTM)

Stuff those in your post-mortems — YAML front matter in a markdown file, if that’s how you roll — and now you have something structured, aggregable, tied to the narrative, and boring in the very best way. MTTD and MTTM stop being feelings and become numbers you can track quarter on quarter. The genuine, considered fix can have its own timestamp later, if you like. It happens at human pace and doesn’t count against your mitigation time.

And break it down by severity. Under the P1–P4 model³, a P1 that affects every user should have a detection time measured in minutes. A P4 affecting Gary in Indiana and his dark mode preferences can wait until Tuesday. Different targets, different alerting, different acceptable MTTM. “Our P1 MTTM is twenty minutes” is a very different sentence from “our P4 MTTM is four hours”, and only one of them should keep you up at night.

Write up all the incidents, too. Especially the small ones. P4s are how you spot the pattern before it grows up into a P1.

The moment you have these numbers, you stop arguing about vibes. If the data says your MTTD is three hours for P1s, you know exactly what you’re working on this quarter.

Postscript One: It’s Not Actually the Mean

Here’s a confession that undermines both acronyms: the mean is the wrong average.

The mean is dragged all over the place by outliers. One horrible six-hour incident and a month of tidy ten-minute ones average out to something that describes none of them. The number is technically true and completely useless.

Use the median and the 95th percentile instead. The median tells you what a normal incident looks like — the Tuesday-afternoon, roll-it-back, home-for-tea incident. The 95th percentile tells you what your nightmares look like — the ones worth actually fixing your process around.

So really it should be MedianTTD and P95TTM, but nobody’s going to say that, and I’m not going to be the one who starts. Just know that when you go to aggregate, mean() is the lazy choice and it lies to you.

Postscript Two: AI Makes This More Urgent

On the subject of AI assistants making us “faster” — and I think we can safely translate “faster” as “we will ship more changes” — there is a corollary. Given our already-professed ignorance, we will also ship more errors. More velocity, more entropy.

MTTF will fall. That’s fine, in itself. But if your MTTM stays where it is, you’ve just added speed to your problems and nothing else. You’ve given yourself a higher error rate and the same slow recovery, which is strictly worse than before.

Optimize for mitigation. It’s always been the right target. The faster you move, the more it matters — and right now, everyone’s moving faster.