You are the on-call rota
There is no handover, so every decision about responding is made once and applies always.
Companies pay people to be woken up, in a rota, so that no individual carries every hour. You have the same responsibility and a rota of one. Whatever you decide about answering, you are deciding it for every hour of every week, including the ones you intend to spend asleep.
Worth getting up for
- Money is moving incorrectly, or not at all
- Data is being lost or exposed
- Nobody can log in
- It is getting worse on its own
Genuinely fine until morning
- One user hit one strange edge case
- A page is slow but working
- Something looks wrong but nothing is wrong
- You have already stopped the bleeding
Then there is the data. A backup you have never restored is not a backup, it is a belief. Restore one into a scratch environment on a calm afternoon, once. You will find out whether it works, and you will learn how long it takes, which is precisely the number you will want to quote to somebody while it matters.
Real users also bring obligations that have nothing to do with code. Somebody will ask you to delete their account and everything attached to it. If the honest answer is that you do not know everywhere their data went, that is worth fixing now, while there are eleven users and three tables, rather than later, under a deadline, with a lawyer reading over your shoulder.
If this broke at 2am on a Sunday, what have I already decided to do about it?
"Nothing until Monday" is a perfectly respectable answer when it is chosen in advance and published. It is a much worse one when it is discovered in the moment by a user who expected otherwise.
What to remember
- Write down what you promise to respond to, and publish it.
- An untested backup is a belief; restore one on a calm day to find out.
- When nothing was deployed, look outward: expiries, upstreams and quotas.
Terms in this lesson
Show field notes toggles a search param the loader reads. With it off, the slow promise is never created, so nothing streams.