A triage order for when it breaks
Stop the bleeding, then find the cause. In that order, always.
The instinct when something breaks is to understand it. The correct first move is to make it stop. Understanding is valuable and can happen tomorrow; users are being affected now.
- 1. What changed?
Almost always a recent deploy, a config change, or a dependency. Start there. - 2. Undo it
Roll back. Do not debug forward with users watching. - 3. Confirm recovery
Verify with real traffic, not with hope. - 4. Now investigate
Calmly, with the pressure off.
Two things make this dramatically easier and both are decided in advance. Deploy small changes, so "what changed" has a short answer. And know how to roll back before you need to. An untested rollback path is not a rollback path.
Afterwards, write down what happened without assigning blame. The useful question is never "who did this" but "what allowed this to reach users", and the answer is usually a missing check that is cheap to add once you have seen why it matters.
What to remember
- Stop the impact before understanding the cause.
- "What changed recently" answers most incidents.
- Small deploys and a practised rollback are what make this work.
Terms in this lesson
Show field notes toggles a search param the loader reads. With it off, the slow promise is never created, so nothing streams.