Logs, metrics, and traces
Three tools that answer three different questions. Most people only have one.
When something goes wrong, the difference between a five minute fix and a five hour one is whether you can see what happened. Three kinds of visibility exist, and they are not substitutes for each other.
Logs, what happened
- Individual events with detail
- Great for one specific failure
- Terrible for spotting trends
- Expensive at volume
Metrics, how much, how often
- Numbers over time: requests, errors, duration
- Great for "is it worse than yesterday"
- Cheap to keep for a long time
- Cannot tell you about one user
The third is tracing: following one request across every service it touches, with timings for each hop. When a page is slow and four systems are involved, a trace tells you which one in seconds. Without it you are guessing across four sets of logs with mismatched clocks.
If this broke right now, what would I actually look at?
If the honest answer is "I would redeploy and hope", that is worth fixing on a calm day rather than discovering on a bad one.
What to remember
- Logs are events, metrics are trends, traces are one request across systems.
- A shared request id makes logs searchable instead of scattered.
- Never log raw request bodies.
Terms in this lesson
Field notes
Loaded from a deliberately slow source. The lesson above was already readable while this was still travelling. That is streaming, and it is the same trick a chat interface uses.
The digest that stopped for five weeks
A weekly email job silently stopped running after an infrastructure change. Nobody noticed, because a job that does not run produces no error. It was discovered when a customer asked whether they had been unsubscribed.
Why you alert on missing success
Everyone retried at once
A service had a brief wobble. Every client retried immediately, then again, then again. The retries were far more traffic than the original load, and the service never got a quiet moment to recover. The outage lasted forty minutes longer than the fault did.
The thundering herd
resolved in 900ms · region iad1
Hide field notes toggles a search param the loader reads. With it off, the slow promise is never created, so nothing streams.