TheCeleryBugThatQuietlyRunsYourTasksTwiceatScale

The Silent Double Execution Problem

There's a specific kind of bug that's particularly dangerous in distributed systems — the kind that hides completely during development, passes all your tests, and only shows up when real users are pounding your application. Double task execution in Celery is exactly that kind of bug. I've watched it silently corrupt data, send users duplicate emails, and charge customers twice before anyone even noticed something was wrong.

The frustrating part is that Celery isn't broken. It's actually doing exactly what it was designed to do. The problem lives in a subtle interaction between how message brokers handle acknowledgements (basically a "receipt" that tells the broker a task was received and processed), how workers behave under high load, and some default configuration choices that make sense for simple setups but quietly fall apart at scale.

Understanding this bug requires understanding a bit about how Celery actually works under the hood. When your application sends a task, Celery places a message in a queue managed by a broker — usually Redis or RabbitMQ. A worker picks up that message, processes the task, and sends back an acknowledgement telling the broker "yep, got it, handled it, you can remove this message." The bug lives in the timing of that acknowledgement.

Why High Load Makes Everything Worse

Think of it like ordering food at a busy restaurant. During a quiet Tuesday lunch, the waiter writes down your order, confirms it back to you, and walks to the kitchen. Clean, simple. But during Saturday dinner rush with a packed house? The waiter might grab your order slip, get pulled away to handle three other tables, and by the time they look at their notepad again, they're not sure if they actually confirmed your order or just picked it up. So they read it again. You get two burgers.

Celery's default acknowledgement mode is called acks_late = False, which means the worker acknowledges the message as soon as it receives it, before the task even starts running. This sounds safe, but it creates a different problem — if the worker crashes or gets killed after acknowledging but before finishing, the task is lost completely. To prevent that, many teams switch to acks_late = True, which delays acknowledgement until after the task completes. This sounds safer, and in many ways it is. But it's also where the double execution bug starts lurking.

With acks_late = True, if a worker picks up a task, starts running it, but then gets interrupted — maybe the Kubernetes pod gets evicted, maybe the process hits a memory limit, maybe the machine restarts — the broker never received an acknowledgement. So it puts the message back in the queue. Another worker picks it up. The task runs again. Under normal load, this is rare and often acceptable. Under high load, with workers being stretched and infrastructure under stress, it happens constantly.

The Visibility Timeout Trap

Here's where it gets sneaky, especially with Redis as your broker. Redis doesn't have native message queue semantics the way RabbitMQ does. Celery works around this using a mechanism called the visibility timeout — a window of time during which a message is "invisible" to other workers after being picked up. If the worker finishes and acknowledges within that window, great. If the window expires before the task finishes, Redis assumes the worker failed and makes the message visible again.

The default visibility timeout in Celery with Redis is one hour. That sounds like plenty of time. And for most tasks, it is. But consider what happens during a traffic spike when you have long-running tasks suddenly queued up behind thousands of shorter ones. Workers are busy. Tasks that normally finish in seconds start waiting minutes in the queue first. Then they finally get picked up by an overloaded worker, and the task itself takes longer than usual because the system is under strain. I've seen tasks that normally finish in 45 minutes creep past the 60-minute mark under load — right past that visibility timeout.

When the timeout expires, Redis doesn't know your task is still running. It just sees an unacknowledged message and makes it visible again. A second worker grabs it and starts from the beginning. Now you have two workers running the same task simultaneously. No error. No warning. Just silent duplication.

# In your Celery configuration (celeryconfig.py or Django settings)
# This is what many teams set without realizing the implications

CELERY_BROKER_TRANSPORT_OPTIONS = {
    'visibility_timeout': 3600  # 1 hour — the default
}

# If your tasks can run longer than this under load,
# you WILL get duplicate execution. Increase it:
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'visibility_timeout': 43200  # 12 hours — safer for long-running tasks
}

The fix here is straightforward once you know about it: set your visibility timeout to be significantly longer than your longest possible task. But this only addresses one piece of the puzzle.

Worker Prefetching Makes It Worse

Celery workers have another behavior called prefetching — they grab multiple messages from the queue at once, before they're actually ready to process them. The idea is to reduce latency by having work ready to go. The default CELERYD_PREFETCH_MULTIPLIER is 4, meaning each worker process can hold 4 tasks in reserve. This is fine when tasks are fast and uniform. It becomes a problem when tasks vary wildly in duration.

Imagine a worker process grabs 4 tasks. It starts working on task 1, which turns out to be a monster — maybe it's generating a report for a huge dataset. Tasks 2, 3, and 4 are just sitting in that worker's local buffer, not being processed, but also not visible to other workers. They're effectively held hostage. From the broker's perspective, they might not be acknowledged either, depending on your configuration, which can trigger the visibility timeout issue on those waiting tasks too.

I've found that setting CELERYD_PREFETCH_MULTIPLIER = 1 is almost always the right call for production systems with mixed task types. Yes, it slightly increases the overhead of fetching from the broker. But it means tasks get distributed fairly across workers, long tasks don't block short ones from other workers, and the visibility timeout behavior becomes much more predictable.

# celery.py or settings.py

app.conf.update(
    # Each worker only takes one task at a time from the queue
    CELERYD_PREFETCH_MULTIPLIER=1,
    
    # Acknowledge AFTER the task runs (safer for reliability)
    CELERY_ACKS_LATE=True,
    
    # Make sure tasks are re-queued if a worker dies unexpectedly
    CELERY_REJECT_ON_WORKER_LOST=True,
    
    # Extend the visibility timeout significantly
    BROKER_TRANSPORT_OPTIONS={
        'visibility_timeout': 43200,  # 12 hours
    }
)

Setting CELERY_REJECT_ON_WORKER_LOST=True is another often-missed configuration. Without it, if a worker process crashes mid-task, the task can end up in a weird limbo state — not acknowledged, but also not explicitly rejected. This setting makes sure that a task is explicitly put back in the queue if the worker handling it dies, rather than silently disappearing or being retried in an unpredictable way.

Making Tasks Idempotent — Your Real Defense

Configuration fixes reduce the probability of double execution significantly. But in distributed systems, you should never rely on "the infrastructure won't do this twice" as your only line of defense. Networks are unreliable. Processes die. Bugs exist in brokers. The professional approach is making your tasks idempotent — a fancy word for "running this twice produces the same result as running it once."

Think about an ATM machine. It's designed so that if the network drops right after you withdraw money but before the bank confirms it, the bank's system can safely re-process that transaction request without charging you twice. The system checks "did this withdrawal already happen?" before executing. Your Celery tasks should work the same way.

The simplest implementation is tracking task execution using a unique task ID and checking that ID before doing any meaningful work. Celery gives every task a unique ID automatically via self.request.id. You can store these IDs in Redis with a short expiry window and bail out early if you see the same ID come through twice.

import redis
from celery import shared_task

redis_client = redis.Redis(host='localhost', port=6379, db=0)

@shared_task(bind=True, acks_late=True)
def send_welcome_email(self, user_id: int):
    # Create a unique lock key for this specific task execution
    lock_key = f"task_lock:{self.name}:{user_id}"
    task_id_key = f"task_done:{self.request.id}"
    
    # Check if this exact task ID has already been processed
    if redis_client.exists(task_id_key):
        # Task already ran successfully — skip it silently
        return {"status": "skipped", "reason": "already_processed"}
    
    # Check if another worker is currently handling this user's email
    # nx=True means "only set if not exists" — atomic operation
    lock_acquired = redis_client.set(lock_key, self.request.id, nx=True, ex=300)
    
    if not lock_acquired:
        # Another worker is handling this — raise to retry later
        raise self.retry(countdown=30, max_retries=3)
    
    try:
        # Do the actual work here
        user = fetch_user(user_id)
        send_email(user.email, "Welcome!", render_welcome_template(user))
        
        # Mark this task ID as successfully completed
        # Expire after 24 hours to keep Redis clean
        redis_client.setex(task_id_key, 86400, "done")
        
        return {"status": "success", "user_id": user_id}
    finally:
        # Always release the lock, even if something goes wrong
        redis_client.delete(lock_key)

This pattern gives you two layers of protection. The task ID check prevents the exact same task instance from running twice — useful when visibility timeout causes a requeue. The user-level lock prevents two different task instances targeting the same user from running simultaneously — useful if a bug in your application accidentally enqueues the same logical work twice.

Monitoring Is Half the Battle

You can't fix what you can't see. And the most insidious thing about this bug is that without proper monitoring, you might go weeks without realizing tasks are running twice. The symptoms show up elsewhere — duplicate records in the database, users complaining about double charges, mysterious spikes in third-party API usage. The cause doesn't look like an error at all.

Set up monitoring specifically for duplicate task execution. One practical approach is logging a structured event at the start and end of every task, including the task ID and any business-level identifiers (like user ID or order ID). Ship these logs to a system like Datadog, Elasticsearch, or even just a database table you can query. Then run periodic checks looking for task IDs that appear in your "started" logs more than once, or business identifiers that appear more than expected within a time window.

Celery also exposes metrics through its celery.monitor module and integrations with tools like Flower (a web-based monitoring tool for Celery) and Prometheus. I've found Flower to be invaluable for seeing worker state in real time — you can watch tasks move through different states and quickly spot if the same task ID shows up in "active" state on two different workers simultaneously, which is a dead giveaway that something is wrong.

Understanding Your Broker Matters

RabbitMQ and Redis behave differently, and those differences matter when debugging this problem. RabbitMQ has proper message queue semantics built in — it tracks which consumer has which message, handles acknowledgements natively at the protocol level, and has more sophisticated dead letter queue support (a separate queue where failed messages go for inspection rather than disappearing). The visibility timeout problem is specific to Redis because Redis is fundamentally a data store being used as a queue, not a purpose-built message broker.

This doesn't mean Redis is the wrong choice. Redis is fast, operationally simpler, and works great for high-throughput task queues. But you need to understand its limitations and configure around them. If you're regularly running tasks that take longer than an hour, and your infrastructure sometimes kills long-running processes, RabbitMQ might actually be the better choice for your situation — not because Redis is bad, but because RabbitMQ's acknowledgement model is more naturally suited to that pattern.

The configuration settings, idempotency patterns, and monitoring approaches described here apply regardless of which broker you use. The specific triggers differ, but the underlying truth is the same: in distributed systems at scale, anything that can happen will happen, and "my task ran twice" is something that can absolutely happen. Building your system to handle it gracefully, rather than assuming it won't occur, is what separates a production-grade task queue implementation from one that's quietly corrupting your data at 3 AM on your highest-traffic day of the year.