RedisTTLDriftIsBreakingYourRateLimitsWithoutWarning

The Silent Killer in Your Rate Limiter

Most engineers I've worked with assume Redis TTLs are precise. Set a key to expire in 60 seconds, and it expires in 60 seconds. Clean. Predictable. Done. But that assumption is quietly destroying rate limiters in production systems every single day, and the worst part is that the logs stay clean while it happens.

TTL drift is the phenomenon where the actual expiration time of a Redis key diverges from what you intended. It's not dramatic. It doesn't throw errors. Your monitoring dashboards look fine. But under the hood, users are either getting throttled when they shouldn't be, or — far more dangerously — bypassing your rate limits entirely because keys are living longer than expected. I've found this bug lurking in production systems at companies that had been running Redis for years without ever suspecting it.

The analogy I always use is a parking meter that runs slow. You pay for an hour, but the meter only counts 50 minutes. The meter reads "expired" when you still have 10 minutes left. Or worse, it runs fast and lets the next person's car sit in your paid spot before your time is actually up. Either way, someone's getting a ticket they don't deserve, or someone's getting away with something they shouldn't.

How Redis Actually Handles Expiry

Before we can understand why TTLs drift, we need to understand how Redis manages expiration internally. Redis doesn't run a background process that continuously scans every key to check if it's expired. That would be prohibitively expensive at scale. Instead, it uses a combination of lazy expiration and active expiration.

Lazy expiration means a key is only checked when you try to access it. If you run GET mykey, Redis checks the expiry timestamp at that moment. If the TTL has passed, Redis deletes the key right then and returns nil. The key was technically "dead" for a while before you touched it, but it still occupied memory. Active expiration runs periodically — by default, Redis samples a random subset of keys with TTLs every 100ms and removes the expired ones it finds. This means that between active expiry cycles, dead keys linger.

Why does this matter for rate limiting? Because your rate limiter logic almost certainly assumes that when a key disappears, the rate limit window has reset. If a key is supposed to expire at T+60s but Redis's active expiry cycle hasn't cleaned it up yet, and your code reads that key at T+61s via a GET command, you'll get nil (lazy expiry kicks in). But if another code path checks via EXISTS and the active cycle just hasn't run yet — in older patterns or buggy implementations — you might still see the key. The timing is non-deterministic. That's the crack in the foundation.

Where the Drift Actually Comes From

The drift has multiple sources, and they compound each other. The most common one I've seen in real systems is clock skew across a Redis cluster. In a Redis Cluster setup (where data is sharded across multiple nodes) or even in a simple primary-replica setup, each machine has its own system clock. Redis uses the local system clock to evaluate TTLs. If one node's clock is 2 seconds behind another, a key might expire 2 seconds later on that node than you intended.

Network Time Protocol (NTP) synchronization is supposed to solve this, but NTP adjustments aren't instantaneous. In virtualized environments like AWS EC2 or Kubernetes pods, clock drift can be surprisingly aggressive. I've measured drift of up to 500ms on busy Kubernetes nodes during peak CPU load, which is plenty to corrupt short-window rate limits. If your window is 1 second or even 10 seconds, a 500ms drift is enormous in relative terms.

The second source is Redis's own internal timer precision. Redis measures time in milliseconds, so theoretically you have millisecond-level TTL granularity with PEXPIRE (the millisecond version of EXPIRE). But the active expiry cycle runs every 100ms by default (hz 10 in the config), which means even perfect keys might sit expired in memory for up to 100ms before removal. Combine this with network latency between your application and Redis, and you have a fuzzy expiry window of potentially 100-200ms even in ideal conditions.

How This Breaks Rate Limiting in Practice

Let me show you a classic sliding window rate limiter that looks correct but has a drift vulnerability baked right in.

import redis
import time

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

def is_rate_limited(user_id: str, limit: int = 100, window_seconds: int = 60) -> bool:
    key = f"rate_limit:{user_id}"
    current_time = time.time()
    
    pipe = r.pipeline()
    # Remove entries outside the current window
    pipe.zremrangebyscore(key, 0, current_time - window_seconds)
    # Add the current request
    pipe.zadd(key, {str(current_time): current_time})
    # Count requests in the window
    pipe.zcard(key)
    # Set TTL on the key
    pipe.expire(key, window_seconds)
    results = pipe.execute()
    
    request_count = results[2]
    return request_count > limit

This pattern is very common and the drift problem is subtle. The expire(key, window_seconds) call sets a TTL of 60 seconds from now. But "now" is determined by the Redis server's local clock when it processes that command. If you're calling this from an application server whose clock is slightly ahead of the Redis server's clock, the current_time you're using to clean up old entries (zremrangebyscore) is ahead of the time Redis uses to measure the TTL. The window boundaries are misaligned. Users near the threshold might get a "you're rate limited" response when they're actually still within their allowed count, because stale entries from outside the true window haven't been cleaned up yet.

The fix is to anchor all time operations to Redis's clock using TIME command or use Lua scripts that run atomically server-side, keeping everything consistent to a single clock source.

import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

SLIDING_WINDOW_SCRIPT = """
local key = KEYS[1]
local now = tonumber(redis.call('TIME')[1]) * 1000 + math.floor(tonumber(redis.call('TIME')[2]) / 1000)
local window_ms = tonumber(ARGV[1]) * 1000
local limit = tonumber(ARGV[2])
local window_start = now - window_ms

redis.call('ZREMRANGEBYSCORE', key, 0, window_start)
redis.call('ZADD', key, now, now)
redis.call('PEXPIRE', key, window_ms)

local count = redis.call('ZCARD', key)
return count
"""

sliding_window = r.register_script(SLIDING_WINDOW_SCRIPT)

def is_rate_limited(user_id: str, limit: int = 100, window_seconds: int = 60) -> bool:
    key = f"rate_limit:{user_id}"
    count = sliding_window(keys=[key], args=[window_seconds, limit])
    return int(count) > limit

By using redis.call('TIME') inside the Lua script, every single operation — cleanup, insertion, TTL — is anchored to the same Redis server clock tick. There's no client clock involved. The script executes atomically, so there's no window for another request to slip in between operations. This alone eliminates the majority of TTL drift issues I've encountered.

The Replica Lag Problem Nobody Talks About

Even with perfectly synchronized clocks, there's another source of TTL corruption that hits you when you scale reads to replica nodes. Redis replication is asynchronous by default. When your primary sets a key with a TTL, that key and its TTL metadata are replicated to replicas — but with a delay. That delay is usually small, but it's never zero.

Here's the nasty part: replicas do not independently manage TTLs in the same way as the primary. In older Redis versions (before 7.0), a replica would not expire keys autonomously by default — it relied on the primary sending explicit DEL commands as keys expired. If your application reads rate limit data from a replica (to reduce primary load), it might see a key that the primary has already logically expired. The replica shows "5 requests in this window" when the primary would show "0 requests, fresh window." Your user just got rate limited when they should have had a fresh quota.

Redis 7.0 improved this with replica-side expiry for read commands, but if you're on an older version — and many production systems are — you need to either read exclusively from the primary for rate limit checks or accept that replica reads introduce expiry uncertainty. I always recommend routing rate limit reads to the primary. The slight increase in primary load is absolutely worth the correctness guarantee. This is one of those cases where sacrificing some read scalability is the only correct engineering choice.

Detecting Drift in Your Running System

You can't fix what you can't measure. Here's a quick diagnostic approach to check whether your Redis instance has meaningful clock drift relative to your application servers.

const Redis = require('ioredis');
const redis = new Redis({ host: 'localhost', port: 6379 });

async function measureClockDrift() {
  const samples = [];
  
  for (let i = 0; i < 20; i++) {
    const clientBefore = Date.now();
    const [seconds, microseconds] = await redis.time();
    const clientAfter = Date.now();
    
    // Redis TIME returns [unix_seconds, microseconds]
    const redisMs = (parseInt(seconds) * 1000) + Math.floor(parseInt(microseconds) / 1000);
    const clientMidpoint = Math.floor((clientBefore + clientAfter) / 2);
    
    const drift = clientMidpoint - redisMs;
    samples.push(drift);
    
    await new Promise(resolve => setTimeout(resolve, 100));
  }
  
  const avgDrift = samples.reduce((a, b) => a + b, 0) / samples.length;
  const maxDrift = Math.max(...samples.map(Math.abs));
  
  console.log(`Average clock drift: ${avgDrift.toFixed(2)}ms`);
  console.log(`Max observed drift: ${maxDrift.toFixed(2)}ms`);
  
  if (maxDrift > 50) {
    console.warn('WARNING: Significant clock drift detected. TTL accuracy may be compromised.');
  }
}

measureClockDrift().catch(console.error);

Run this script from each of your application servers. Any drift over 50ms is worth investigating. Over 200ms and you have an active problem that's affecting users right now. The network round-trip time will add some noise to the measurement, which is why taking 20 samples and using the midpoint approximation gives you a more accurate picture than a single measurement.

Fixing the Configuration Layer

Beyond code changes, there are Redis configuration settings that reduce the window for drift-related issues. The hz parameter controls how often Redis runs its active expiry cycle — it defaults to 10 (every 100ms). Increasing it to 25 or even 100 makes expiry more timely, at the cost of slightly higher CPU usage from the Redis process itself. For rate limiting workloads where key expiry precision matters, this is usually a worthwhile trade.

# In redis.conf or via redis-cli
CONFIG SET hz 25

# For even tighter expiry enforcement (higher CPU cost)
CONFIG SET hz 100

# Also enable dynamic hz, which scales hz based on connected clients
CONFIG SET dynamic-hz yes

The dynamic-hz setting is particularly useful — it lets Redis increase the frequency of background tasks automatically when more clients are connected and there's more work to do, while backing off during quiet periods. I've seen this alone cut the "zombie key" window (expired keys still in memory) in half on busy rate-limiting clusters.

You should also set lazyfree-lazy-expire yes in your Redis config. This moves the actual memory reclamation for expired keys to a background thread, which prevents expiry operations from blocking the main Redis event loop. When Redis is under heavy load, blocking expiry can delay the processing of subsequent commands by tens of milliseconds, which indirectly causes drift in timestamp comparisons.

The Fixed Window Alternative and Its Own Drift Problem

Some teams switch to fixed window rate limiting to avoid the complexity of sliding windows, but the TTL drift problem hits them differently and arguably harder. A fixed window rate limiter creates a new key at the start of each time window, increments a counter, and relies entirely on the key expiring at the window boundary to reset the count.

The classic implementation looks innocent: INCR rate:user:123 followed by EXPIRE rate:user:123 60. But this has a well-known race condition where the EXPIRE never gets set if the process crashes between the two commands. The correct approach uses SET key 0 EX 60 NX to create the key with a TTL atomically, then INCR. Even then, if the key expires 50ms late due to active expiry cycle timing, a user who hammers requests right at the window boundary gets two full windows of quota almost simultaneously — their counter resets right as they're maxing out the old window. At 100 requests per minute, that means 200 requests go through in under a second.

The window-boundary problem is particularly brutal because it's not random — it's exploitable. A determined user (or a botnet) can learn your rate limit window boundaries and time attacks specifically to exploit the expiry gap. I've seen this pattern in API abuse logs more than once. The sliding window approach, while more complex, is inherently more resistant to this because there's no single "reset moment" to target.

Putting It All Together: A Robust Rate Limiter Architecture

The pragmatic solution combines several layers. Use Lua scripts to keep all time operations server-side. Route rate limit reads and writes to the primary node, not replicas. Set hz 25 or higher in your Redis config. Use PEXPIRE with millisecond precision instead of EXPIRE. And critically, build in a small buffer to your TTL — if your window is 60 seconds, set the TTL to 63 seconds. This ensures the key outlives its useful window by enough margin to absorb active expiry cycle delays, preventing the "ghost window" scenario where a key expires before you've finished using it.

Monitoring matters too. Track the actual measured TTLs of your rate limit keys using PTTL and compare them to your expected values. Alert when drift exceeds your tolerance threshold. In my experience, most teams discover TTL drift only after a security incident or a customer complaint about being wrongly throttled — that's the worst possible time to learn about it.

The underlying truth is that Redis TTLs are probabilistic, not deterministic, when you account for real-world operating conditions. Building a rate limiter that assumes deterministic expiry is like building a bridge that assumes perfect materials with zero tolerance — it works fine until the real world shows up. Design for the drift, measure it continuously, and your rate limiter will actually do what you need it to do.