TheSilentDataLossHidingInsideYourWebhookRestartTiming

The Problem Nobody Talks About Until It's Too Late

There's a specific kind of pain that comes from debugging missing data. Not corrupted data, not wrong data — missing data. The kind where your database says one thing, your provider's dashboard says another, and you spend three hours staring at logs that show absolutely nothing wrong because the problem happened in a gap that left no trace.

I've spent more hours than I care to admit chasing this exact scenario, and the culprit is almost always the same: a webhook event that arrived at the worst possible moment — right in the middle of a server restart. Your server was shutting down. The request was in flight. And somewhere between the TCP handshake and the final database write, the whole thing just... vanished.

This is silent data loss. And the insidious part is that your monitoring won't catch it. Your error tracking won't fire. Your logs will look clean. The webhook provider sent the event, got a response (or didn't, and gave up retrying), and moved on. Your system never knew what it missed.

What Actually Happens During a Server Restart

To understand why this happens, you need a mental model of what a server restart actually looks like at the network level. Think of your server like a busy restaurant kitchen. Orders (requests) are constantly coming in, and cooks (worker processes) are preparing them. When the restaurant decides to close for cleaning, there's a right way and a wrong way to do it. The wrong way is to shut off all the lights and lock the doors immediately — customers with half-cooked meals are just left hanging.

That's essentially what happens when your deployment pipeline sends a SIGTERM signal (the polite "please stop now" signal that operating systems use) to your process and your app doesn't handle it gracefully. The OS starts shutting things down. Active HTTP connections get interrupted. Any webhook payload that was in the middle of being processed — parsed, validated, written to a queue, saved to a database — just stops. The function call stack unwinds. The data disappears.

The problem gets worse when you factor in how webhook providers behave. Most major providers like Stripe, GitHub, and Twilio have retry logic, but it varies wildly. Stripe, for instance, will retry a failed webhook over the course of 72 hours, but it considers any non-2xx response as a failure — including a connection reset or a timeout. If your server restarted mid-request and the connection was dropped at the TCP layer before any HTTP response was sent, the provider might log it as a delivery failure and retry. Or it might not. Depending on timing, it could have received a RST packet (a network-level "connection reset" signal) and logged it as delivered. This ambiguity is where data goes to die.

The Anatomy of a Dropped Webhook Event

Let me walk through the exact sequence of events that causes this. You have a Node.js or Python server handling Stripe payment webhooks. A customer just completed a checkout. Stripe fires a checkout.session.completed event to your endpoint. At the exact same moment, your CI/CD pipeline is deploying a new version of your app and sends SIGTERM to the current process.

Here's what the timeline looks like:

1. Stripe opens a TCP connection to your server
2. Your server accepts the connection (the process is still running)
3. Stripe sends the HTTP POST request with the event payload
4. Your server starts reading the request body
5. SIGTERM arrives
6. Your server (without graceful shutdown) immediately begins cleanup
7. The in-flight request is abandoned mid-read
8. The TCP connection is dropped or reset
9. Stripe receives an error or timeout
10. Depending on Stripe's retry logic and timing, this event may or may not be retried successfully

The gap between steps 5 and 7 is where your data vanishes. And if you're using a process manager like PM2 without configuring kill_timeout, or if your Kubernetes pod has a terminationGracePeriodSeconds that's too short, this window exists every single time you deploy.

The Code That Makes You Vulnerable

Here's a typical Express.js webhook handler that looks perfectly fine but is completely vulnerable to this issue:

const express = require('express');
const app = express();

app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
  const sig = req.headers['stripe-signature'];
  let event;

  try {
    event = stripe.webhooks.constructEvent(req.body, sig, process.env.STRIPE_WEBHOOK_SECRET);
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  // This is where the real work happens
  if (event.type === 'checkout.session.completed') {
    const session = event.data.object;
    await fulfillOrder(session); // Database write, email send, etc.
  }

  res.json({ received: true });
});

const server = app.listen(3000);

This code has no signal handling whatsoever. When SIGTERM arrives, Node.js will start its cleanup process, and fulfillOrder() might never finish. There's no acknowledgment back to Stripe, and depending on what fulfillOrder was doing, your database could be in a partial state.

Building a Gracefully Shutting Server

The fix requires two things working together: a server that finishes its in-flight requests before dying, and a webhook processing architecture that doesn't rely solely on the HTTP response cycle for data durability. Let's tackle the first one.

const express = require('express');
const app = express();

// Track active connections
let isShuttingDown = false;

app.use((req, res, next) => {
  if (isShuttingDown) {
    res.setHeader('Connection', 'close');
    res.status(503).send('Server is shutting down');
    return;
  }
  next();
});

app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
  const sig = req.headers['stripe-signature'];
  let event;

  try {
    event = stripe.webhooks.constructEvent(req.body, sig, process.env.STRIPE_WEBHOOK_SECRET);
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  if (event.type === 'checkout.session.completed') {
    const session = event.data.object;
    await fulfillOrder(session);
  }

  res.json({ received: true });
});

const server = app.listen(3000, () => {
  console.log('Server running on port 3000');
});

// Graceful shutdown handler
const shutdown = (signal) => {
  console.log(`${signal} received. Starting graceful shutdown...`);
  isShuttingDown = true;

  // Stop accepting new connections
  server.close((err) => {
    if (err) {
      console.error('Error during server close:', err);
      process.exit(1);
    }
    console.log('All connections closed. Shutting down cleanly.');
    process.exit(0);
  });

  // Force shutdown after 30 seconds if connections don't close
  setTimeout(() => {
    console.error('Forced shutdown after timeout');
    process.exit(1);
  }, 30000);
};

process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));

The key insight here is server.close(). This method stops the server from accepting new connections but allows existing connections to finish naturally. The middleware that checks isShuttingDown handles any new requests that sneak in via keep-alive connections — they get a 503 response telling the client to try again somewhere else. The 30-second timeout is your safety net, preventing the process from hanging forever if something goes wrong during shutdown.

Why Graceful Shutdown Alone Isn't Enough

Here's the uncomfortable truth: graceful shutdown reduces the window for data loss, but it doesn't eliminate it. If your fulfillOrder function takes 45 seconds because it's sending emails, resizing images, and talking to three external APIs, and your shutdown timeout is 30 seconds, you're still going to lose data. The process will be killed before the work is done.

Think of it like that restaurant analogy again. Graceful shutdown means you stopped seating new customers and you're letting existing orders finish. But if the chef needs 45 minutes to finish a complicated dish and the health inspector says you have 30 minutes to clear out, someone's meal is still getting abandoned.

The real solution is to decouple the work from the HTTP response cycle entirely. This is where a message queue comes in. A message queue (like Redis with BullMQ, RabbitMQ, or AWS SQS) is a separate, durable storage system that can hold your events safely even if your application crashes. Instead of processing the webhook synchronously in the HTTP handler, you immediately store the raw event data and respond to the provider, then process the data separately.

import json
import hmac
import hashlib
from flask import Flask, request, jsonify, abort
import redis
from rq import Queue  # Redis Queue - a job queue library for Python

app = Flask(__name__)

# Connect to Redis
redis_conn = redis.Redis(host='localhost', port=6379, db=0)
webhook_queue = Queue('webhooks', connection=redis_conn)

def process_stripe_event(event_data):
    """This runs in a separate worker process, safely away from HTTP lifecycle"""
    event_type = event_data.get('type')
    
    if event_type == 'checkout.session.completed':
        session = event_data['data']['object']
        fulfill_order(session)
    elif event_type == 'customer.subscription.deleted':
        subscription = event_data['data']['object']
        cancel_subscription(subscription)
    
    # Add more event types as needed

def verify_stripe_signature(payload, sig_header, secret):
    """Verify the webhook actually came from Stripe"""
    try:
        timestamp, signature = None, None
        for item in sig_header.split(','):
            key, value = item.strip().split('=', 1)
            if key == 't':
                timestamp = value
            elif key == 'v1':
                signature = value
        
        signed_payload = f"{timestamp}.{payload.decode('utf-8')}"
        expected_sig = hmac.new(
            secret.encode('utf-8'),
            signed_payload.encode('utf-8'),
            hashlib.sha256
        ).hexdigest()
        
        return hmac.compare_digest(expected_sig, signature)
    except Exception:
        return False

@app.route('/webhooks/stripe', methods=['POST'])
def stripe_webhook():
    payload = request.get_data()
    sig_header = request.headers.get('Stripe-Signature', '')
    
    if not verify_stripe_signature(payload, sig_header, app.config['STRIPE_WEBHOOK_SECRET']):
        abort(400, 'Invalid signature')
    
    try:
        event_data = json.loads(payload)
    except json.JSONDecodeError:
        abort(400, 'Invalid JSON')
    
    # Store the raw event in Redis for idempotency tracking
    event_id = event_data.get('id')
    already_processed = redis_conn.get(f"webhook:processed:{event_id}")
    
    if already_processed:
        # Already queued or processed - safe to acknowledge and skip
        return jsonify({'received': True, 'status': 'duplicate'})
    
    # Mark as received BEFORE queuing to prevent race conditions
    redis_conn.setex(f"webhook:processed:{event_id}", 86400, 'queued')  # 24h TTL
    
    # Enqueue for background processing - this is fast and durable
    webhook_queue.enqueue(process_stripe_event, event_data)
    
    # Respond immediately - our work here is done
    return jsonify({'received': True})

This architecture is fundamentally safer. The HTTP handler does two things: validates the signature and drops the event on a queue. Both operations are fast — we're talking milliseconds. The actual work happens in a separate worker process that can crash, restart, and retry without affecting whether Stripe got its 200 response. Even if your web server crashes immediately after responding, the event is already safely sitting in Redis.

Idempotency: Your Safety Net for Retry Chaos

Even with graceful shutdown and a message queue, you need to deal with the reality that webhook providers will sometimes send the same event more than once. This can happen because of network hiccups, their own retry logic, or just the fundamental messiness of distributed systems. The concept you need is idempotency — making your processing logic safe to run multiple times with the same input and always produce the same result.

I've found that the simplest approach is to use the webhook's event ID as a database key. Before processing any event, check if you've seen that ID before. If you have, skip it. If you haven't, process it and record the ID. This one-two punch of "check then record" needs to be atomic (meaning it happens as a single uninterruptible operation) to avoid race conditions when two copies of the same event arrive simultaneously.

The Python example above shows a simple Redis-based approach to this. In a production system, you might store processed event IDs in your main database instead, which gives you a permanent audit trail. The tradeoff is slightly more latency on each webhook request, which is almost always worth it for the peace of mind.

Infrastructure-Level Protections

The code changes matter, but they need infrastructure backing them up. If you're running on Kubernetes, you need to configure your pod's terminationGracePeriodSeconds to be longer than your longest possible request processing time. The default is 30 seconds, which sounds reasonable until you have a webhook handler that talks to a slow third-party API. A value of 60 to 120 seconds is more realistic for most production workloads.

# In your Kubernetes deployment manifest
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60  # Give pods 60 seconds to finish
      containers:
        - name: your-app
          lifecycle:
            preStop:
              exec:
                # Add a small sleep to let the load balancer drain connections
                # before SIGTERM is sent to the process
                command: ["/bin/sleep", "5"]

The preStop hook is a subtle but important addition. When Kubernetes decides to terminate a pod, it simultaneously sends SIGTERM and removes the pod from the load balancer's list of healthy endpoints. But "simultaneously" in Kubernetes terms has some lag — the load balancer update can take a few seconds to propagate. During those few seconds, the load balancer might still send new requests to a pod that's already received SIGTERM. The preStop sleep gives the load balancer time to catch up before your graceful shutdown logic starts refusing new connections.

If you're using PM2 as a process manager, the equivalent configuration lives in your ecosystem.config.js:

module.exports = {
  apps: [{
    name: 'webhook-server',
    script: 'server.js',
    kill_timeout: 30000,       // Wait 30s before force-killing
    wait_ready: true,          // Wait for process.send('ready') signal
    listen_timeout: 10000,     // How long to wait for 'ready'
    shutdown_with_message: true // Use 'shutdown' message instead of SIGINT
  }]
};

Monitoring the Gap You Can't See

One of the hardest parts of this problem is building visibility into something that, by its nature, leaves minimal traces. You need to actively instrument your system to surface these gaps rather than waiting for a customer to complain about a missing order.

Most webhook providers have a dashboard showing event delivery history. Set up a scheduled job — I'd recommend every hour — that queries your provider's API for recent events and cross-references them against your local records. Any event that your provider says was delivered but doesn't appear in your database is a red flag. Stripe's API, for example, lets you list all events with their delivery status, and this kind of reconciliation query can surface silent failures that would otherwise go unnoticed for days.

Combine this with application-level metrics on your webhook handler. Track not just success and failure rates, but also the timing distribution of your handler's response times. If you see a spike in p99 (99th percentile) latencies during deployment windows, that's a signal that long-running requests are competing with your shutdown logic. That pattern, even if it's not causing data loss today, is telling you that you're one bad timing coincidence away from a problem.

The goal is to make the invisible visible before it costs you. Silent data loss is patient. It will wait until the worst possible moment — peak traffic, a critical customer, a promotional campaign — to show up. Building the observability and infrastructure to catch it early is one of those unglamorous engineering investments that pays for itself the moment something goes wrong.