The Night Everything Lied to Me
It was 2 AM, and our monitoring dashboard was showing green. All systems nominal. Redis Sentinel — the high-availability watchdog that's supposed to keep our caching layer alive — was happily reporting that our primary node was up and healthy. Customers were throwing requests at our API. And then the alerts started. Slowly at first, then all at once: stale session data, double-processed payments, users getting logged out mid-transaction.
The primary wasn't healthy. It hadn't been for eleven minutes. But Sentinel was still cheerfully pointing clients at a node that had quietly partitioned away from the rest of the cluster. We had a split-brain scenario — two nodes simultaneously believing they were the leader — and nobody told us until the damage was done. I've spent a lot of time since that night understanding exactly how this happens, because the answer is more subtle and unsettling than "the network went down."
What Split-Brain Actually Means
Let me use an analogy before we get into the technical weeds. Imagine a country with one president. The president flies to a remote island for a summit, and a massive storm cuts off all communication between the island and the mainland. The mainland government, hearing nothing for days, assumes the president is dead and swears in a new one. But the original president is alive and fine — just cut off. Now you have two people simultaneously issuing executive orders, signing contradictory laws, and making decisions that conflict with each other. When communication is restored, the country is in chaos. That's split-brain.
In Redis terms, split-brain (sometimes called network partition) happens when your Redis nodes lose the ability to communicate with each other, but each partition continues operating independently. One side elects a new primary. The original primary keeps accepting writes. You now have two sources of truth, and they are diverging with every passing second. When the partition heals, one side's data gets thrown away — and whichever side loses the election takes all its recent writes to the garbage bin.
The thing people don't appreciate is that split-brain isn't a rare, catastrophic failure. Network hiccups happen constantly in cloud environments. A flaky switch, a misconfigured security group, a noisy neighbor saturating a shared NIC — any of these can create the conditions for a momentary partition that's long enough for Sentinel to panic and promote a new primary, but short enough that your original primary never stops accepting writes.
How Redis Sentinel Is Supposed to Work
Redis Sentinel is a separate process — actually a collection of separate processes — that monitors your Redis nodes and orchestrates failover when the primary becomes unavailable. You're supposed to run at least three Sentinel instances for a meaningful quorum (quorum means the minimum number of Sentinels that have to agree before any action is taken). The idea is that you need a majority to agree before promoting a replica to primary, which should prevent rogue decisions during network partitions.
Here's the basic lifecycle. A Sentinel notices the primary isn't responding. It marks it as Subjectively Down (SDOWN) — just that one Sentinel's opinion. If enough Sentinels agree, the node gets marked Objectively Down (ODOWN). Then the Sentinels elect a leader among themselves using a modified Raft consensus algorithm. That leader Sentinel performs the failover: it picks the best replica, promotes it, and tells all other replicas to follow the new primary. Then it reconfigures clients using a notification mechanism. Clean, right?
The problem is that "enough Sentinels agree" is configurable, and misconfigured quorums are everywhere. The default quorum value in many tutorial setups is 1, which means a single Sentinel deciding the primary is down is enough to trigger a failover. One Sentinel. One bad network packet. One moment of CPU saturation on the primary causing a slow PING response. That's all it takes. And here's where the lying starts — the original primary isn't actually down. It's alive, it's serving traffic, it just looked slow for a second. But Sentinel has already moved on and promoted a new leader.
The Min-Replicas Trap Nobody Talks About
There's a Redis configuration parameter called min-replicas-to-write (formerly min-slaves-to-write) that is your single most important defense against split-brain data loss, and I've audited dozens of production Redis setups where it's set to zero. Zero means "the primary will accept writes even if it has zero connected replicas." That's the default. That is insane for any system where you care about your data.
What this parameter does is refuse write commands on the primary if fewer than the specified number of replicas are connected and in sync. Think about what that means during a partition. If your primary gets partitioned away from its replicas, it can no longer replicate data. With min-replicas-to-write 1, the primary would detect this, stop accepting writes, and essentially self-demote to a read-only state. Meanwhile, Sentinel promotes a replica on the other side. When the partition heals, the original primary rejoins as a replica with no divergent writes to reconcile. Clean failover. No split-brain data loss.
The companion setting min-replicas-max-lag controls how stale the replica can be before it "doesn't count." Set min-replicas-max-lag 10 to say "I only count replicas that are within 10 seconds of the primary." Together these two settings create a fence — a mechanism where the primary voluntarily fences itself out of the write path when it's isolated. Without this fence, your primary keeps happily accepting writes into the void, and you're setting yourself up for exactly the 2 AM scenario I described.
# In your redis.conf for the PRIMARY node
min-replicas-to-write 1
min-replicas-max-lag 10
# In your sentinel.conf
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
# Quorum of 2 means at least 2 of your 3 sentinels must agreeThe down-after-milliseconds value also deserves attention. Many guides suggest 30000 (30 seconds), but I've found that in practice you want this lower — around 5000ms — to reduce the window during which writes go to a partitioned primary. The tradeoff is more false positives on flaky networks. Tune it based on your actual network characteristics.
How Clients Make Everything Worse
Here's a dimension of the problem that even experienced engineers miss: your application clients have their own connection pools, their own retry logic, and their own cached knowledge of who the primary is. Even after Sentinel successfully promotes a new primary and notifies the world, your clients may be stubbornly talking to the old primary for minutes. Some client libraries cache the primary's address and only re-discover it on connection failure. Others respect Sentinel notifications immediately. The difference matters enormously.
The standard approach to Redis with Sentinel is to connect to Sentinel, ask it who the primary is, then connect directly to that primary. This means your client does a two-step: first SENTINEL get-master-addr-by-name <name> to Sentinel, then connects to whatever address Sentinel returns. The problem is that this lookup happens at connection time, and connection pools reuse connections. If your pool opened a connection to the old primary before failover, it will keep using that connection until it breaks.
The right pattern is to use a client library that truly supports Sentinel-aware connection management. In Python with redis-py, this looks like the following:
import redis
from redis.sentinel import Sentinel
# Connect to your Sentinel instances, not directly to Redis
sentinel = Sentinel(
[('sentinel-host-1', 26379),
('sentinel-host-2', 26379),
('sentinel-host-3', 26379)],
socket_timeout=0.5, # Fail fast if Sentinel is unresponsive
password='your-sentinel-password' # If you've secured Sentinel
)
# This gives you a smart connection that re-queries Sentinel on failure
primary = sentinel.master_for(
'mymaster',
socket_timeout=0.5,
password='your-redis-password',
decode_responses=True
)
replica = sentinel.slave_for(
'mymaster',
socket_timeout=0.5,
password='your-redis-password',
decode_responses=True
)
# Writes go to primary, reads can go to replica
try:
primary.set('session:12345', 'user_data', ex=3600)
value = replica.get('session:12345')
except redis.exceptions.ReadOnlyError:
# This is your signal that you're talking to a demoted primary
# The library should reconnect to the new primary automatically
print("Caught ReadOnly error — Sentinel failover probably in progress")
primary = sentinel.master_for('mymaster', socket_timeout=0.5)The ReadOnlyError catch is critical. When Sentinel demotes the old primary to a replica, Redis changes that node to read-only mode. If your client sends a write to it, you get READONLY You can't write against a read only replica. This error is your application's last chance to know something went wrong. If you're swallowing generic exceptions, you might be silently discarding writes for minutes.
The Fencing Token Pattern for Critical Writes
For operations where data loss is genuinely unacceptable — think financial transactions, inventory updates, anything with real-world side effects — Sentinel alone isn't enough. Distributed systems researchers have a concept called fencing tokens, and it's worth understanding even if you don't implement it directly.
The core idea is this: every time a new primary is elected, it gets a monotonically increasing epoch number (a counter that only ever goes up). When your application performs a write, it includes this epoch number. The Redis node rejects any write that arrives with an epoch older than its current one. This way, an isolated zombie primary that hasn't received the memo about the new election will reject writes from any client that has discovered the new epoch.
Redis doesn't natively support fencing tokens for arbitrary operations, but you can approximate this pattern using Lua scripts that atomically check and update an epoch key:
-- This Lua script runs atomically on the Redis server
-- It checks the current epoch before writing, ensuring we're talking to the right primary
local current_epoch = redis.call('GET', 'cluster:epoch')
local client_epoch = ARGV[1]
local key = KEYS[1]
local value = ARGV[2]
local ttl = ARGV[3]
if current_epoch == false then
-- First write, initialize the epoch
redis.call('SET', 'cluster:epoch', client_epoch)
elseif tonumber(current_epoch) > tonumber(client_epoch) then
-- Our epoch is stale — we're probably talking to an old primary
return redis.error_reply('ERR_STALE_EPOCH: client epoch ' .. client_epoch .. ' behind cluster epoch ' .. current_epoch)
end
redis.call('SET', key, value, 'EX', ttl)
return 'OK'// Node.js example calling the fencing token script
const redis = require('ioredis');
const client = new redis({
sentinels: [
{ host: 'sentinel-1', port: 26379 },
{ host: 'sentinel-2', port: 26379 },
{ host: 'sentinel-3', port: 26379 }
],
name: 'mymaster',
password: 'your-redis-password'
});
// Load the script and get its SHA for efficient reuse
const fencingScript = `
local current_epoch = redis.call('GET', 'cluster:epoch')
local client_epoch = ARGV[1]
local key = KEYS[1]
local value = ARGV[2]
local ttl = ARGV[3]
if current_epoch == false then
redis.call('SET', 'cluster:epoch', client_epoch)
elseif tonumber(current_epoch) > tonumber(client_epoch) then
return redis.error_reply('ERR_STALE_EPOCH')
end
redis.call('SET', key, value, 'EX', ttl)
return 'OK'
`;
async function safeWrite(key, value, epoch, ttl = 3600) {
try {
const result = await client.eval(fencingScript, 1, key, epoch, value, ttl);
return result;
} catch (err) {
if (err.message.includes('ERR_STALE_EPOCH')) {
// We're talking to a stale primary — trigger reconnection
console.error('Stale epoch detected. Possible split-brain condition.');
// Implement your escalation logic here
throw new Error('SPLIT_BRAIN_DETECTED');
}
throw err;
}
}This pattern won't save you from every split-brain scenario, but it creates a tripwire. When the partitioned cluster heals and you have two primaries with divergent epochs, the fencing logic starts throwing errors rather than silently accepting contradictory writes.
Testing Your Setup Before It Breaks in Production
The only way to know if your Sentinel configuration actually handles split-brain correctly is to deliberately create split-brain conditions in a test environment. I've seen teams spend months configuring Sentinel beautifully in theory, then never test it, and discover the gaps only when production is on fire.
The iptables command on Linux is your friend here. You can drop network traffic between specific hosts to simulate partitions without actually powering anything off. This is far more realistic than just killing a Redis process, because a dead process fails fast and clean — a partitioned process is the zombie that keeps on writing.
# Simulate a network partition between primary Redis and its replicas
# Run this on the PRIMARY node
sudo iptables -A INPUT -s <replica-ip-1> -j DROP
sudo iptables -A OUTPUT -d <replica-ip-1> -j DROP
sudo iptables -A INPUT -s <sentinel-ip-1> -j DROP
sudo iptables -A OUTPUT -d <sentinel-ip-1> -j DROP
# Watch what happens in your application — check for ReadOnlyErrors,
# check your monitoring, check if the epoch changes
# ...
# When you're ready to heal the partition
sudo iptables -F
# Check what happened to data on both sides
redis-cli -h <old-primary> DEBUG SLEEP 0 # just to ping it
redis-cli -h <new-primary> INFO replicationRun this during a load test and watch your metrics carefully. Did writes pause while Sentinel was electing a new leader? Good — that means min-replicas-to-write is working. Did you see any ReadOnlyError exceptions in your application logs? Good — that means your client caught the failover. Did both sides accept writes during the partition window? That's the problem, and you need to know about it before your users do.
In my experience, most teams are shocked to discover how long the partition window actually is. With default Sentinel settings, you can easily have a 30-45 second window where the old primary is still accepting writes after it's been partitioned. That's 30-45 seconds of data that will be lost when the partition heals and the demoted primary syncs from the new one. Whether that's acceptable depends entirely on your application, but you should make that decision consciously, not accidentally.
The Bigger Truth About Eventual Consistency
Underneath all of this, there's a fundamental truth about Redis Sentinel that you need to internalize: Sentinel is not a strongly consistent system. It's an eventually consistent, best-effort high-availability mechanism. Redis itself, being primarily an in-memory store with asynchronous replication, trades consistency for performance. Writes are acknowledged before they're replicated. This is fine for caches. It is not fine for source-of-truth data stores, which people absolutely use Redis as.
The CAP theorem (a foundational principle in distributed systems that says you can only guarantee two of: Consistency, Availability, and Partition Tolerance) tells us that during a network partition, we must choose between consistency and availability. Redis Sentinel chooses availability. It keeps serving reads and writes during a partition, at the cost of potentially having inconsistent data on different sides. This is a design decision, not a bug. But it becomes a bug when engineers don't understand it and design systems that assume Redis with Sentinel is as consistent as Postgres with synchronous replication.
If you need true strong consistency with Redis-level performance, you need to look at Redis Cluster with WAIT commands to enforce synchronous replication on critical writes, or you need to accept that your application must be designed to tolerate data loss in the failover window. The WAIT command (blocks the client until the specified number of replicas have acknowledged the write, up to a timeout) is a blunt instrument but an effective one for critical writes: WAIT 1 100 says "block until at least 1 replica confirms this write, or give up after 100ms." This dramatically reduces the data loss window at the cost of some latency.
What I Changed After That Night
After that 2 AM incident, we made four concrete changes. We set min-replicas-to-write 1 and min-replicas-max-lag 10 on every primary. We increased our Sentinel quorum from 1 to 2 (we run three Sentinels). We added explicit handling for ReadOnlyError in every Redis client throughout our codebase, treating it as an urgent signal rather than a generic error. And we added a monthly chaos testing session where we deliberately partition our staging Redis cluster and verify that our application handles it gracefully.
None of these changes are complicated. The configuration changes are literally two lines in a config file. But they require understanding why you're making them, which means understanding how split-brain happens and how Sentinel's optimism can become your liability. Sentinel isn't lying to you out of malice — it's working exactly as designed. The design just has failure modes that aren't obvious until 2 AM when the alerts are firing and you're trying to explain to your CEO why some customers got charged twice.
The lesson isn't "don't use Redis Sentinel." Sentinel is a solid, battle-tested system when configured correctly and used for the right use cases. The lesson is that high availability tools don't make distributed systems easy — they make them possible, with a whole new set of failure modes you have to understand, test, and design around. The split-brain bug I almost shipped was never really about Redis. It was about my assumptions, and how long I'd left them untested.