Designing a Task Queue System

A task queue processes background jobs asynchronously — image processing, email sending, report generation.

Why Task Queues?

• Offload slow operations from request path
• Retry failed jobs
• Schedule future jobs
• Rate-limit expensive operations

Architecture

API Server → Queue (Redis/RabbitMQ) → Workers

Worker 1: Process image
Worker 2: Send email
Worker 3: Generate report

Job Lifecycle

Pending → Running → Success
                  ↘
                  Failed → Retry
                         → Dead Letter Queue (max retries exceeded)

Priority Queues

Multiple queues with different priorities:

high_priority: payment_jobs
normal: email_jobs
low: report_jobs

Workers check high-priority queue first.

Retry Strategy

Exponential backoff:

Attempt 1: retry in 1 min
Attempt 2: retry in 2 min
Attempt 3: retry in 4 min
Attempt 4: retry in 8 min
Attempt 5: → Dead Letter Queue

Delayed Jobs

# Schedule job for future
queue.schedule(send_reminder, run_at=datetime.now() + timedelta(hours=24))

Implement with Redis ZADD (sorted by timestamp), poll with cron.

Job Deduplication

Prevent same job from being enqueued twice:

hash(job_type + args) → Idempotency key in Redis

Monitoring

Track per queue:

• Queue depth
• Job throughput
• Failed job rate
• Worker utilization

Conclusion

Redis + worker processes (Celery/Sidekiq pattern) is the battle-tested approach. Add dead letter queues and exponential backoff for reliability.