The accounts you now have
Nobody decided to sign up for seven services. It happened one problem at a time.
Count them. Somewhere you have a host, a database, an authentication provider, something that sends email, a domain registrar, probably an error tracker, and a model API. Seven logins, most with a card attached, all belonging to one person. Nobody sat down and chose this. Each one was the fastest way past a problem on the day it appeared.
Fixed cost, sleeps quietly
- A flat monthly fee, agreed in advance
- Going viral does not change the number
- Worst case is that it stops working
- Easy to reason about, easy to forget
Usage-priced, can surprise you
- Per request, per gigabyte, per token
- A bug and a busy night cost the same as success
- Worst case is an invoice, not an outage
- This is where the frightening stories come from
The defence is dull and takes minutes: set a hard spending cap where the vendor offers one, and a billing alert where it does not. Do it on the day you create the account, not the day you need it, because the day you need it you are asleep and the meter is not.
- Which accounts exist, and which card each one bills.
- Which are usage-priced, and whether each has a cap set.
- Which keys those accounts issued, and where each one lives.
- Who besides you can log in, including the assistant you pasted a key into.
- What visibly breaks if any single one of them is switched off.
If ten thousand people visited tonight, which of my accounts could send me a bill I cannot pay?
You are looking for the ones priced per use with no cap set. There are usually one or two, they are usually the model API and the host, and both take about five minutes to bound.
What to remember
- Fixed-cost services fail by stopping; usage-priced services fail by charging.
- Caps and billing alerts are worth setting the day an account is created.
- A leaked key is fixed by issuing a replacement first, then retiring the old one.
Terms in this lesson
Field notes
Loaded from a deliberately slow source. The lesson above was already readable while this was still travelling. That is streaming, and it is the same trick a chat interface uses.
The digest that stopped for five weeks
A weekly email job silently stopped running after an infrastructure change. Nobody noticed, because a job that does not run produces no error. It was discovered when a customer asked whether they had been unsubscribed.
Why you alert on missing success
Everyone retried at once
A service had a brief wobble. Every client retried immediately, then again, then again. The retries were far more traffic than the original load, and the service never got a quiet moment to recover. The outage lasted forty minutes longer than the fault did.
The thundering herd
resolved in 900ms · region iad1
Hide field notes toggles a search param the loader reads. With it off, the slow promise is never created, so nothing streams.