Queues and workers
A list of things to do, and processes that pull from it. That is genuinely all it is.
A queue is a durable list of work waiting to be done. A worker is a program that repeatedly takes an item, does it, and marks it finished. Everything else in the topic is detail hanging off those two sentences.
- Producers put work on the queue, usually your web app, during a request.
- The queue holds it durably, so a crash does not lose it.
- Workers pull items off and do them, at whatever rate they can manage.
- More workers means faster drainage, and you can add them without touching your app.
Because the queue survives crashes and the workers are separate, you can deploy your web app without interrupting jobs in flight, and you can scale the two independently. That decoupling is why queues show up in every system that has been running for more than a year.
Where do failed jobs go?
A job that fails forever and retries forever is a slow-motion outage. Mature setups move repeat failures to a dead letter queue, a holding pen someone actually looks at.
What to remember
- A queue is a durable to-do list; a worker is a process that drains it.
- Queues absorb spikes by delaying work rather than dropping it.
- Jobs will sometimes run more than once. Design for that.
Terms in this lesson
Show field notes toggles a search param the loader reads. With it off, the slow promise is never created, so nothing streams.