A triage order for when it breaks
Stop the bleeding, then find the cause. In that order, always.
The instinct when something breaks is to understand it. The correct first move is to make it stop. Understanding is valuable and can happen tomorrow; users are being affected now.
- 1. What changed?
Almost always a recent deploy, a config change, or a dependency. Start there. - 2. Undo it
Roll back. Do not debug forward with users watching. - 3. Confirm recovery
Verify with real traffic, not with hope. - 4. Now investigate
Calmly, with the pressure off.
Two things make this dramatically easier and both are decided in advance. Deploy small changes, so "what changed" has a short answer. And know how to roll back before you need to. An untested rollback path is not a rollback path.
Afterwards, write down what happened without assigning blame. The useful question is never "who did this" but "what allowed this to reach users", and the answer is usually a missing check that is cheap to add once you have seen why it matters.
What to remember
- Stop the impact before understanding the cause.
- "What changed recently" answers most incidents.
- Small deploys and a practised rollback are what make this work.
Terms in this lesson
Field notes
Loaded from a deliberately slow source. The lesson above was already readable while this was still travelling. That is streaming, and it is the same trick a chat interface uses.
The digest that stopped for five weeks
A weekly email job silently stopped running after an infrastructure change. Nobody noticed, because a job that does not run produces no error. It was discovered when a customer asked whether they had been unsubscribed.
Why you alert on missing success
Everyone retried at once
A service had a brief wobble. Every client retried immediately, then again, then again. The retries were far more traffic than the original load, and the service never got a quiet moment to recover. The outage lasted forty minutes longer than the fault did.
The thundering herd
resolved in 901ms · region iad1
Hide field notes toggles a search param the loader reads. With it off, the slow promise is never created, so nothing streams.