Queues and workers
A list of things to do, and processes that pull from it. That is genuinely all it is.
A queue is a durable list of work waiting to be done. A worker is a program that repeatedly takes an item, does it, and marks it finished. Everything else in the topic is detail hanging off those two sentences.
- Producers put work on the queue, usually your web app, during a request.
- The queue holds it durably, so a crash does not lose it.
- Workers pull items off and do them, at whatever rate they can manage.
- More workers means faster drainage, and you can add them without touching your app.
Because the queue survives crashes and the workers are separate, you can deploy your web app without interrupting jobs in flight, and you can scale the two independently. That decoupling is why queues show up in every system that has been running for more than a year.
Where do failed jobs go?
A job that fails forever and retries forever is a slow-motion outage. Mature setups move repeat failures to a dead letter queue, a holding pen someone actually looks at.
What to remember
- A queue is a durable to-do list; a worker is a process that drains it.
- Queues absorb spikes by delaying work rather than dropping it.
- Jobs will sometimes run more than once. Design for that.
Terms in this lesson
Field notes
Loaded from a deliberately slow source. The lesson above was already readable while this was still travelling. That is streaming, and it is the same trick a chat interface uses.
The digest that stopped for five weeks
A weekly email job silently stopped running after an infrastructure change. Nobody noticed, because a job that does not run produces no error. It was discovered when a customer asked whether they had been unsubscribed.
Why you alert on missing success
Everyone retried at once
A service had a brief wobble. Every client retried immediately, then again, then again. The retries were far more traffic than the original load, and the service never got a quiet moment to recover. The outage lasted forty minutes longer than the fault did.
The thundering herd
resolved in 900ms · region iad1
Hide field notes toggles a search param the loader reads. With it off, the slow promise is never created, so nothing streams.