DockerWorksLocally,DiesinProd:OOMKillerandcgroupSecrets

The Mystery That Haunts Every DevOps Engineer

You've been there. Your Docker container runs perfectly on your laptop. Tests pass, the app responds quickly, memory looks fine. You push to production, deploy the container, and within hours — sometimes minutes — everything falls apart. The app crashes silently, requests time out, or the container just restarts with no useful error message. You dig through logs and find nothing. Absolutely nothing.

This isn't bad luck. It's not a flaky test environment. It's a predictable, well-understood problem that stems from the gap between your local Docker setup and how containers actually run in production environments with resource constraints. The culprit is almost always the OOM Killer — the Linux kernel's mechanism for dealing with memory exhaustion — combined with cgroup limits (control groups, which are the kernel feature that lets Linux isolate and limit resources for a group of processes). Together, these two things can murder your container silently, and your application logs will never breathe a word about it.

I've spent more late nights than I care to admit staring at a wall of empty log output wondering why a service that ran flawlessly for hours locally fell over in production after ten minutes. Once I understood what was actually happening at the kernel level, everything clicked. Let me show you what's really going on.

What cgroups Actually Are (And Why You Should Care)

Think of cgroups like a landlord setting rules for an apartment building. Each tenant (container) gets a unit with defined limits — only so much electricity (CPU), only so much storage space (disk), and only so much water (memory). Your local development machine is like living in your own house with no landlord. You can use as much electricity as you want because nobody's watching. But in a production Kubernetes cluster or a managed container platform, the landlord is very much present and very strict.

cgroups are the Linux kernel feature that Docker, Kubernetes, and every other container runtime uses to enforce these limits. When you run a container with --memory=512m, the kernel creates a cgroup with a hard memory ceiling of 512 megabytes. Cross that ceiling, and the kernel doesn't politely ask your process to use less memory. It takes more drastic action.

Here's where most people miss the critical detail: on your local machine, you probably run Docker without any memory limits at all. Your container has access to your full 16GB or 32GB of RAM. In production, that same container might be constrained to 512MB or 1GB. The application behaves completely differently under those constraints, and the failure mode is brutal and silent.

The OOM Killer: Your Application's Silent Executioner

The OOM Killer (Out Of Memory Killer) is a subsystem in the Linux kernel that activates when a system or a cgroup runs out of memory. Its job is simple and merciless: pick a process and kill it to free up memory. It's like a lifeboat with too many passengers — someone has to go overboard, and the kernel makes that decision algorithmically.

The OOM Killer scores each process based on factors like how much memory it's using, how long it's been running, and whether it's marked as critical. The process with the highest "badness score" gets signal 9 — SIGKILL — sent directly from the kernel. No warning. No graceful shutdown. No chance to write a final log message. The process just stops existing, and if that process is your Node.js server or your Python web worker, the container exits with code 137 (which is 128 + 9, representing a SIGKILL).

I've found that most developers see exit code 137 and assume it means "something crashed." They don't realize it almost always means the OOM Killer specifically terminated the process. The difference is enormous for debugging. A crash might mean a bug in your code. An OOM kill means your application is consuming more memory than the container is allowed to have — and that's an infrastructure and application design problem, not necessarily a bug.

Why Your Logs Are Silent

This is the part that drives people absolutely crazy. When the OOM Killer fires, it doesn't log to your application. It can't. The kernel kills the process before it has any opportunity to write a log entry, flush a buffer, or gracefully shut down. Your application framework's crash handler never runs. Your custom error handlers are bypassed completely. The process simply ceases.

Think of it like a power outage mid-sentence while you're writing a letter. The letter stops wherever your pen was. There's no "P.S. — by the way, the power went out." The evidence of what happened exists in the kernel ring buffer (the kernel's own internal log), not in your application logs. Most application monitoring setups don't capture kernel-level events, which means the murder weapon is literally invisible to your observability stack.

To actually see what happened, you need to look at the right places. On a Linux host, you can check the kernel messages with dmesg. In a Kubernetes environment, you look at the container's last state. Let me show you both.

# Check kernel messages for OOM events on the host
sudo dmesg | grep -i "oom\|killed process\|out of memory"

# Check a container's previous termination reason in Kubernetes
kubectl describe pod <your-pod-name> -n <namespace>

# Look specifically at the last state of a container
kubectl get pod <your-pod-name> -o jsonpath='{.status.containerStatuses[0].lastState}'

# Check container exit code — 137 almost always means OOM kill
docker inspect <container-id> --format='{{.State.ExitCode}}'

That kubectl describe pod output will show you a lastState section with reason: OOMKilled if the container was terminated by the OOM Killer. This is the smoking gun most people never look for because they're too busy tailing application logs that contain nothing useful.

Reproducing the Problem Locally (So You Can Actually Fix It)

The most powerful thing you can do is reproduce the production constraints on your local machine. This sounds obvious, but almost nobody does it. We run Docker locally with zero resource limits and then wonder why behavior differs in production. Close the gap by explicitly imposing the same limits your production environment uses.

Here's a realistic example. Suppose your production container has 512MB of memory and 0.5 CPUs allocated. Run it locally like this:

# Run your container with production-matching resource limits
docker run \
  --memory=512m \
  --memory-swap=512m \
  --cpus=0.5 \
  --name my-app-constrained \
  my-app:latest

# In a separate terminal, watch memory usage in real time
docker stats my-app-constrained

# After a crash, check the exit code
docker inspect my-app-constrained --format='{{.State.ExitCode}} - {{.State.Error}}'

Setting --memory-swap equal to --memory disables swap entirely, which matches how most production Kubernetes environments behave. Swap is typically disabled on Kubernetes nodes for performance reasons, so if your application is relying on the generous swap space available on your dev machine to handle memory spikes, it's going to fail hard in production where swap doesn't exist.

A Real-World Node.js Memory Leak That OOM Kills in Production

Let me walk through a concrete scenario. Here's a simplified Node.js Express server that has a classic memory leak — it caches responses in memory indefinitely. Locally, with 16GB of RAM, it runs for hours without any issue. In production with a 256MB container limit, it dies within minutes of a moderate traffic load.

const express = require('express');
const app = express();

// This is the leak — responseCache grows forever and is never cleared
const responseCache = new Map();

app.get('/data/:id', async (req, res) => {
  const { id } = req.params;

  if (responseCache.has(id)) {
    return res.json(responseCache.get(id));
  }

  // Simulate fetching data — each "record" is about 10KB
  const data = {
    id,
    payload: Buffer.alloc(10240).toString('base64'), // ~10KB of data
    timestamp: Date.now(),
    metadata: Array.from({ length: 100 }, (_, i) => ({ key: i, value: `item_${i}` }))
  };

  // Store in cache but NEVER evict — this is the problem
  responseCache.set(id, data);

  res.json(data);
});

// A monitoring endpoint that shows current memory usage
app.get('/health', (req, res) => {
  const used = process.memoryUsage();
  res.json({
    status: 'ok',
    memory: {
      heapUsed: `${Math.round(used.heapUsed / 1024 / 1024)}MB`,
      heapTotal: `${Math.round(used.heapTotal / 1024 / 1024)}MB`,
      rss: `${Math.round(used.rss / 1024 / 1024)}MB`,
      cacheSize: responseCache.size
    }
  });
});

app.listen(3000, () => console.log('Server running on port 3000'));

Every unique request ID adds approximately 10KB to the cache. Hit this endpoint with 25,000 unique IDs and you've consumed roughly 250MB just in the cache. On your local machine, Node.js happily allocates more heap. In a 256MB container, the OOM Killer steps in and the container exits with code 137. No log entry in your application output. Just silence, then a restart.

The fix involves implementing proper cache eviction — using an LRU (Least Recently Used) cache that automatically drops old entries, or setting a maximum size. But you'd never find this problem without simulating production memory constraints locally.

cgroup v2 and the New Subtleties

Linux has two versions of the cgroup interface — cgroup v1 and cgroup v2. cgroup v2 is the newer unified hierarchy and it's now the default on most modern Linux distributions including Ubuntu 22.04, Fedora, and the distributions underlying newer cloud compute instances. This matters because the memory accounting and limit enforcement behavior changed in ways that can catch you off guard.

In cgroup v1, memory accounting was sometimes inconsistent depending on how kernel memory (the memory used by kernel operations on behalf of your process, like file descriptor tables and socket buffers) was counted. In cgroup v2, kernel memory is always included in the container's memory accounting. This means a container that was just under its limit on an older cgroup v1 system might cross the limit on a newer cgroup v2 system — even with identical application code — because kernel memory usage is now counted against the container's budget.

I've seen this exact issue bite teams who upgraded their Kubernetes node images from Ubuntu 20.04 (cgroup v1) to Ubuntu 22.04 (cgroup v2). The containers started getting OOM-killed on the new nodes despite no code changes. The solution was either to increase memory limits to account for the additional kernel memory overhead (typically 20-50MB per container) or to profile and optimize the application's overall memory footprint. Neither is a quick fix, but understanding why it was happening made the solution clear.

Reading the Signals: Container Restart Patterns as Diagnostics

One of the most reliable indicators of OOM kills in production is the container restart count and the timing of those restarts. Containers killed by the OOM Killer tend to restart on a very predictable schedule — they come up, accept traffic, gradually consume memory under load, hit the limit, get killed, and restart. The cycle repeats with clockwork regularity.

Contrast this with a crash caused by an unhandled exception or a deadlock. Those crashes tend to happen faster, often at startup or under specific input conditions, and the timing is less regular. If your container is restarting every 45 minutes like a metronome, that rhythmic pattern is a strong signal that memory is accumulating gradually until the limit is hit. Track your container's memory usage over time using docker stats locally or your cluster's metrics system in production.

In Kubernetes specifically, you can set up memory usage alerts using Prometheus and Grafana. But even without that tooling, just watching the kubectl top pods output over time tells a compelling story. Memory that climbs steadily and never drops — even between requests — is a leak. Memory that spikes and returns to baseline is normal heap activity. The difference is visible once you know what to look for.

Setting Proper Limits: The Art of Right-Sizing

Many teams either set no resource limits at all (because they're afraid of OOM kills) or copy limits from a tutorial without measuring actual usage. Both approaches are problematic. No limits mean a runaway process can consume all memory on a node and affect every other container running there. Arbitrary limits mean you're either wasting resources or creating artificial failure conditions.

The right approach is to measure first. Run your application under realistic load — load testing with a tool like k6, locust, or wrk — while watching memory consumption with docker stats. Run the load test for at least 30 minutes to catch gradual leaks. Note the peak memory usage under realistic load, then add a 20-30% headroom for safety. That headroom buffer accounts for normal variation, garbage collection pauses where memory temporarily spikes before being freed, and the kernel memory overhead discussed earlier.

# Run a load test against your locally constrained container
# Install wrk first: https://github.com/wg/wrk
wrk -t4 -c100 -d300s http://localhost:3000/data/test

# In a separate terminal, log memory stats every 5 seconds to a file
docker stats my-app-constrained \
  --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}" \
  --no-stream >> memory-log.txt & \
  watch -n 5 'docker stats my-app-constrained --no-stream'

This gives you empirical data to work from rather than guesses. If peak memory under load is 380MB, set your limit to 500MB. Not 256MB because that's what someone else's tutorial suggested, and not unlimited because "we don't want it to crash."

What to Add to Every Container Deployment

After going through the OOM investigation process more times than I'd like, I've settled on a set of practices that I apply to every containerized service I work on. They don't eliminate OOM issues, but they make them visible and debuggable instead of silent and mysterious.

First, always run containers with explicit memory limits in every environment — including local development — so that environment parity exists from day one. Second, expose a /health or /metrics endpoint from your application that reports process memory usage, so your monitoring stack can track it over time. Third, configure your container orchestrator to capture the lastState of terminated containers and alert on OOMKilled reasons.

And most importantly, treat an OOM kill as a first-class incident. When a container gets OOM-killed, something real is wrong — either the application has a memory leak, the limits are set too low for actual usage, or a traffic spike has exposed a scaling gap. The OOM kill is the symptom. The actual problem is one of those three things, and it's always worth tracking down which one. The kernel already did the hard work of detecting the problem. Don't waste that signal by just restarting the container and moving on.

The gap between "works locally" and "dies in production" is almost always a gap in understanding the environment, not a gap in the code itself. Once you understand what cgroups enforce, what the OOM Killer does, and how to observe both — the mystery evaporates and the path to a fix becomes clear.