The Labs / Backend
BackendWorking~6h
A job queue that doesn't lose work
A durable background job queue: producers enqueue jobs, a worker pulls and processes them at-least-once, failures get retried with exponential backoff, and a job that fails N times lands in a dead-letter queue instead of retrying forever or vanishing. You finish with a system that survives a worker crash mid-job and never silently drops work.
What it proves
The line you can defend in an interview.
Résumé line
Built a durable job queue (at-least-once delivery, visibility timeout, exponential backoff, dead-letter queue after N failures) and proved crash-mid-processing never loses a job.
- Understands at-least-once delivery and why idempotent handlers matter downstream
- Designs for the worker crash, not just the happy path
- Knows when to stop retrying and escalate instead of retrying forever
The brief
What you build, step by step.
- 01enqueue(job) persists the job with status=pending and a unique id before returning.
- 02A worker claims a pending job atomically, sets status=processing with a visibility timeout, and never lets two workers claim the same job concurrently.
- 03On success, mark status=done. On failure, increment attempt count and reschedule with exponential backoff (base * 2^attempt, capped).
- 04If a job's attempt count exceeds N (configurable), move it to a dead-letter queue instead of rescheduling.
- 05If a worker dies mid-processing (visibility timeout expires without completion), the job becomes reclaimable by another worker.
- 06Expose queue_depth(), dlq_depth(), and a way to requeue a DLQ job manually.
The proof
It’s done when these pass.
A job whose handler always fails ends up in the DLQ after exactly N attempts, not before and not after
automated test
Retry delays between attempts increase (backoff is actually exponential, not fixed)
automated test
A job whose worker is killed mid-processing (visibility timeout expires) is picked up and completed by a second worker, exactly once counted as success
automated test
Two workers polling concurrently never both claim the same pending job
automated test
Stack
PythonRedis or SQLite (durable store)a worker process
Sage Method
frame → route → map → decide → prove
You keep
A working job queue + worker + DLQ, with a crash-recovery test suite