>Celery, RabbitMQ, and Redis: patterns for background work that can't fail quietly

The moment an HTTP request needs to send an SMS, charge a card, or process an uploaded PDF, it needs to stop being an HTTP request's problem. Background job systems look simple in a tutorial and become genuinely hard the first time a worker crashes mid-task at 3am with a partially-charged customer. These are the patterns that have kept that from turning into an incident.

1. Every task must be idempotent

Workers get killed mid-execution — by deploys, by OOM, by the broker redelivering an unacknowledged message. If a task isn't safe to run twice, it will run twice eventually, and the second run needs to be a no-op, not a double charge or a duplicate SMS.

@shared_task(bind=True, max_retries=5)
def charge_customer(self, order_id: str):
    order = Order.objects.select_for_update().get(id=order_id)
    if order.status == "charged":
        return  # already done — safe to re-run
    ...

The check-before-act pattern above is unglamorous but it's the single highest-leverage line in any payment or notification task.

2. Pick the broker for the delivery guarantee you actually need

• RabbitMQ when tasks route between multiple services and durability matters more than raw throughput — it acknowledges, retries, and dead-letters explicitly, which makes failure modes visible instead of silent.
• Redis when the job is a single service's internal queue and speed matters more than delivery guarantees — cheaper to run, but a crashed Redis without persistence can lose queued jobs.

Mixing them is common and reasonable: RabbitMQ for the durable, cross-service jobs (payments, order fulfillment) and Redis for the disposable, fast ones (cache warms, low-stakes notifications).

3. Structured logging is the only way you'll debug this in production

A background task has no request context to inspect after the fact — no browser tab, no live terminal. Every task should log a correlation ID at start and end, and every retry should log why it retried:

logger.info("task.start", task="charge_customer", order_id=order_id, attempt=self.request.retries)

Without this, a failed task at 3am is a mystery. With it, it's a five-minute log search.

Heavy work: CV and image pipelines

For CPU-heavy jobs — generating a PDF from a CV builder, resizing uploaded images — the same rules apply, plus one more: cap concurrency per worker so a burst of uploads doesn't starve the queue for lightweight jobs like SMS delivery. Separate queues by workload weight, not just by feature, so one slow pipeline can't block time-sensitive notifications behind it.

Takeaway

Idempotency, an honest choice of broker, and structured logging aren't nice-to-haves for background jobs — they're the difference between a queue that fails loudly and one that fails silently. Silent failures are the ones that cost trust.