StopMisconfiguringLivenessProbesBeforeTheyCrashYourBusiestPods

The Silent Killer Hiding in Your Cluster

There's a particular kind of production incident that keeps platform engineers up at night — not the obvious crash, not the memory leak that slowly eats your nodes, but the subtle, maddening pattern where your pods start dying during your highest-traffic moments. Everything looks fine on the surface. Your application is healthy. Your metrics are green. And then, right when load spikes, Kubernetes starts restarting your pods like it's trying to help but is actually making everything worse.

I've found this pattern in more production clusters than I care to admit, and almost every time, the culprit is the same: a misconfigured liveness probe. The irony is brutal. Liveness probes exist to protect your application — to tell Kubernetes "this pod has gone brain-dead, please restart it." But when configured without accounting for real-world load behavior, they become assassins, taking out perfectly healthy pods at the worst possible moment.

Understanding why this happens requires understanding how Kubernetes actually thinks about pod health, and why the default assumptions baked into most probe configurations are dangerously optimistic.

What Liveness Probes Actually Do (And What They Don't)

A liveness probe is Kubernetes asking your pod one simple question on a repeating timer: "Are you still alive?" If your pod answers correctly, life goes on. If it fails to answer — or answers wrong — enough times in a row, Kubernetes declares the pod dead and restarts it. Think of it like a security guard doing rounds. They knock on the door, and if nobody answers after a few tries, they assume something bad has happened inside and call for a reset.

The problem is that "not answering" and "actually dead" are two very different things. A pod under heavy load might be so busy processing requests that it can't respond to the health check in time. A pod performing a garbage collection cycle might pause for a few seconds. A pod that just received a massive burst of traffic might temporarily exhaust its connection pool — not because it's broken, but because it's working. The security guard analogy breaks down here: a healthy employee who's in the middle of a critical phone call shouldn't be dragged out of the building because they didn't open the door fast enough.

The distinction Kubernetes makes between liveness and readiness probes matters enormously here. A readiness probe (which answers "are you ready to accept traffic?") causes traffic to stop routing to the pod. A liveness probe causes the pod to restart. That's an entirely different consequence, and it demands a much more conservative, deliberate configuration strategy.

The Load-Induced Failure Pattern

Here's the scenario I see play out repeatedly. A team deploys an application with a liveness probe that hits an HTTP endpoint, maybe /health or /ping. During normal traffic, this works perfectly. The endpoint responds in under 100 milliseconds, the probe is happy, and nobody thinks twice about it. Then a traffic spike hits — maybe a product launch, maybe a viral moment, maybe just end-of-month processing.

Under load, the application's thread pool fills up. The event loop (if you're running Node.js, for example) gets saturated. Database connection pools hit their limits. The application is processing work — it's doing exactly what it should — but suddenly that /health endpoint has to wait in line behind real work. The probe times out. Then it times out again. And again. After three consecutive failures, Kubernetes kills the pod and starts a new one.

Now here's the catastrophic part: every time a pod restarts, the pods that survive have to absorb even more traffic. Which means they're even more likely to fail their liveness probes. Which means more restarts. Which means fewer pods. I've watched clusters cascade into near-complete unavailability because of this exact feedback loop, starting from a traffic spike that the application could have otherwise handled if Kubernetes had just left it alone.

The Numbers That Bite You: failureThreshold and timeoutSeconds

The default values for liveness probes are where most configurations go wrong. Let's look at what people typically write:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

This looks reasonable. Thirty seconds before the first check, checking every ten seconds, tolerating three failures. But do the math. With failureThreshold: 3 and periodSeconds: 10, your pod can be killed after just 30 seconds of degraded health check response times. Thirty seconds. That's nothing during a traffic spike. Many applications take longer than that just to flush their in-flight requests and warm back up after a restart.

The timeoutSeconds field defaults to 1 second if you don't specify it. One second. Even moderately loaded applications on shared infrastructure can blow past that timeout occasionally. I've seen teams run their probe health endpoint through their full middleware stack — authentication, logging, database pings — and then wonder why probes fail under load. The health endpoint should be lightweight. It should not touch the database. It should not authenticate. It should not trigger any side effects. It should simply confirm that the process is alive and capable of responding.

Here's a more production-hardened configuration that accounts for real-world load behavior:

livenessProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  initialDelaySeconds: 60
  periodSeconds: 30
  failureThreshold: 5
  timeoutSeconds: 5
startupProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

Notice the startupProbe — this is your application's "give me time to boot" mechanism, and it prevents the liveness probe from running at all until startup succeeds. Without it, a slow-starting application might fail liveness checks during initialization, triggering a restart loop before the app ever has a chance to serve traffic. With the numbers above, your pod gets up to 10 minutes to start (30 × 10 seconds), and once running, needs to fail five consecutive 5-second-timeout checks to get restarted. That's a much more forgiving and realistic posture.

Building a Liveness Endpoint That Won't Become a Liability

The implementation of the health endpoint itself matters just as much as the probe configuration. Here's a pattern in Node.js that I've found works well in high-throughput services:

const express = require('express');
const app = express();

// Track application-level state
let isShuttingDown = false;

// Liveness: is the process itself alive and responsive?
// Keep this FAST and dependency-free
app.get('/healthz/live', (req, res) => {
  if (isShuttingDown) {
    return res.status(503).json({ status: 'shutting_down' });
  }
  
  // Just confirm the event loop is responsive
  res.status(200).json({ 
    status: 'alive',
    timestamp: Date.now()
  });
});

// Readiness: is the app ready to serve real traffic?
// This one CAN check dependencies
app.get('/healthz/ready', async (req, res) => {
  try {
    await db.ping(); // Check database connectivity
    await cache.ping(); // Check cache connectivity
    res.status(200).json({ status: 'ready' });
  } catch (err) {
    res.status(503).json({ status: 'not_ready', error: err.message });
  }
});

process.on('SIGTERM', () => {
  isShuttingDown = true;
  // Give in-flight requests time to complete
  setTimeout(() => process.exit(0), 30000);
});

The key separation here is intentional. The liveness endpoint does almost nothing — it just proves the event loop is spinning and the process can respond. The readiness endpoint does the heavier work of checking dependencies. This means that even a pod under severe load can almost always answer the liveness check quickly, because there's no competition for resources on that path.

In Python with FastAPI, the same principle applies:

from fastapi import FastAPI, Response
import asyncio
import time

app = FastAPI()
startup_time = time.time()
_is_healthy = True

@app.get("/healthz/live")
async def liveness():
    """
    Liveness probe endpoint.
    MUST be fast. No I/O. No external calls.
    """
    if not _is_healthy:
        return Response(status_code=503, content="unhealthy")
    
    return {
        "status": "alive",
        "uptime_seconds": int(time.time() - startup_time)
    }

@app.get("/healthz/ready") 
async def readiness(db=Depends(get_db)):
    """
    Readiness probe endpoint.
    Can check dependencies. Will affect traffic routing, not restarts.
    """
    try:
        await db.execute("SELECT 1")
        return {"status": "ready"}
    except Exception as e:
        return Response(
            status_code=503,
            content=f"Database unavailable: {str(e)}"
        )

The Startup Probe: Your Most Underused Tool

Many engineers I talk to haven't actually used startupProbe in production, or don't fully understand why it exists separately from initialDelaySeconds. The difference matters. initialDelaySeconds is a fixed wait — you're just telling Kubernetes to sit on its hands for N seconds before starting to check. It's a blunt instrument. If your application sometimes starts in 20 seconds and sometimes takes 90 seconds (due to schema migrations, large caches warming up, or slow dependency availability), a fixed initial delay will either be too short or wastefully long.

A startupProbe is adaptive. It polls the same endpoint on a configurable interval, and it disables the liveness probe entirely until the startup probe succeeds. So your application can take as long as it legitimately needs, and Kubernetes still knows to stop waiting if the app is truly broken on startup (which is different from taking longer than expected). This is a fundamental improvement in safety for any application with variable initialization time.

In my experience, teams that skip startup probes tend to compensate by setting aggressively high initialDelaySeconds values — like 120 or 180 seconds — which just means genuinely crashed pods sit dead for three minutes before getting restarted. That's trading one problem for another. The startup probe gives you both protection and responsiveness.

Resource Constraints Make Everything Worse

There's another dimension to this problem that doesn't get enough attention: the interaction between liveness probes and resource requests/limits. When you set CPU limits on a pod (using resources.limits.cpu), Kubernetes uses CPU throttling to enforce them. CPU throttling doesn't kill your process — it just slows it down. Significantly. Under heavy load, a CPU-throttled pod might have all the logical capacity to serve your health check, but be physically delayed by the Linux kernel's CPU scheduler.

This is especially dangerous with aggressive CPU limits on JVM-based applications. The JVM's garbage collector is CPU-intensive, and when the collector needs to run but the CPU is throttled, GC pauses can extend dramatically. A 200-millisecond GC pause under normal conditions can become a 2-second pause under CPU throttling. If your timeoutSeconds is set to 1, you'll fail probes during garbage collection — not because your application is unhealthy, but because the kernel is being stingy with CPU time.

The fix isn't necessarily to remove CPU limits (though there's a legitimate argument for being more generous with them). The fix is to set your timeoutSeconds high enough to survive normal GC pauses and scheduling jitter. I typically recommend at least 5 seconds for JVM applications, and sometimes 10 seconds for applications doing complex processing. This is not being soft — this is being accurate about what "unresponsive" actually means for your application.

Separating Probe Failures From Application Failures

One mindset shift that helps enormously is thinking about what you actually want a liveness probe to detect. The probe is not supposed to check whether your application is performing well. It's not supposed to validate that all dependencies are up. It's not supposed to run your integration test suite. It has one job: detect whether the process has entered an unrecoverable state from which it cannot recover without a restart.

Classic examples of genuinely restart-worthy conditions: a deadlock that has frozen all worker threads, a memory corruption that has made the process non-functional, or an unhandled exception that has caused the main serving loop to exit. These are states where a restart genuinely helps. Contrast those with: high latency because of database slowness, temporary connection pool exhaustion, or elevated error rates because an upstream service is flaky. Restarting a pod does nothing to fix those problems — it just adds the overhead of a restart on top of an already stressed situation.

When you approach probe design with this lens, the configuration choices become more obvious. You're trying to catch the process-is-fundamentally-broken case, and you want to be extremely patient about everything else. That patience is expressed through generous timeoutSeconds, a higher failureThreshold, and a periodSeconds value that gives your application room to breathe rather than polling it relentlessly during its hardest moments.

Testing Your Probes Under Realistic Load

The most reliable way to catch probe misconfiguration before it bites you in production is to test probe behavior explicitly under load — not just correctness under normal conditions. Most teams test that their health endpoint returns 200 OK. Almost nobody tests what happens to probe response times when the application is at 80% CPU utilization.

Load testing tools like k6, Vegeta, or even hey can hammer your application while you simultaneously watch probe metrics in your observability stack. In Prometheus, the kube_pod_container_status_restarts_total metric combined with timestamps from your load test will reveal immediately whether probes are triggering restarts during high-traffic periods. Kubernetes events (kubectl get events --sort-by='.metadata.creationTimestamp') will also show liveness probe failures as they happen.

Simulating CPU throttling intentionally is also useful. You can temporarily set very tight CPU limits on a test deployment and see how your health endpoint response time behaves under throttling. If it climbs above your timeoutSeconds threshold, you have your answer before production tells you the hard way. This kind of pre-production chaos testing around probe behavior is one of the higher-value investments a platform team can make, and it costs surprisingly little time compared to the incidents it prevents.