The Day My Logs Vanished Into Thin Air
I was three hours into debugging a production incident when I realized my logs weren't there. Not corrupted. Not truncated. Just gone. The application had crashed under a spike of traffic, and the last 200 lines of logs I desperately needed had evaporated somewhere between my console.log calls and the log aggregator. That was the day I stopped trusting Node.js logging without understanding what was actually happening underneath.
This is a problem that bites engineers at the worst possible time — during an outage, under load, right when you need visibility the most. The frustrating part is that everything looks fine during development. Your logs stream beautifully to the terminal. You add logging statements, you see them, you trust them. Then you deploy, traffic picks up, your process crashes or gets killed, and suddenly a chunk of your logs simply never existed. Understanding why this happens requires understanding three things: how process.stdout actually works, what buffering means in this context, and how Node.js handles streams when the environment changes.
What process.stdout Actually Is
Most developers think of process.stdout as a simple pipe. You write something to it, it comes out the other end. That mental model is close enough for simple cases, but it misses a critical detail: process.stdout is a stream, and streams in Node.js have different behaviors depending on what they're connected to.
A stream, in plain terms, is like a garden hose. Data flows through it, but the pressure and speed at which it flows can be controlled. Node.js streams can operate in different modes, and process.stdout specifically behaves differently depending on what it's connected to on the other end — a terminal (TTY, short for "TeleTYpewriter," a direct interactive console), or a pipe or file. When you run your app locally and your stdout is connected to your terminal, it runs in synchronous mode. Every console.log call blocks until the data is written. You can see your logs appear immediately, in order, reliably.
But when your app runs in a production environment — behind a process manager like PM2, inside a Docker container, piped through a log collector — stdout is connected to a pipe or a file, not a terminal. In that case, Node.js switches to asynchronous, buffered mode. The write calls become non-blocking. Node queues up the data, and it gets flushed to the destination in batches. This is actually faster and more efficient for high-throughput scenarios. The problem is that if your process exits before that buffer gets flushed, the buffered data disappears forever.
The Buffering Trap Explained
Think of buffering like a waiter taking orders at a busy restaurant. Instead of running to the kitchen after every single customer request, the waiter holds onto several orders and delivers them all at once. This is efficient. But if the restaurant catches fire and everyone evacuates immediately, those orders the waiter was holding? Gone. Nobody got their food.
That's exactly what happens when Node.js crashes, gets killed with SIGTERM or SIGKILL, or even exits normally while there's still data sitting in the stdout buffer. The operating system doesn't care about your pending write operations when a process terminates abruptly. It just cleans up the file descriptors and moves on. Your carefully crafted log messages about the error that caused the crash — the ones that would tell you exactly what went wrong — were sitting in that buffer, waiting to be flushed, and they got discarded with the process.
The insidious thing is that this only happens under certain conditions. Low traffic, local development, small log volumes — you'll never see it. It's only when writes pile up faster than they can be flushed, or when a crash interrupts an in-progress batch flush, that logs disappear. I've seen teams spend days adding more logging to debug a production issue, getting increasingly frustrated that their new log statements weren't showing up, completely unaware that the problem was buffering, not their logging logic.
Reproducing the Problem
Let's make this concrete with some code. Here's a simple script that demonstrates the issue:
// lossy-logs.js
const http = require('http');
const server = http.createServer((req, res) => {
// Simulate some work and logging
for (let i = 0; i < 100; i++) {
console.log(`Request received: ${req.url} - log line ${i}`);
}
res.end('OK');
});
server.listen(3000, () => {
console.log('Server running on port 3000');
});
// Simulate a sudden crash after 2 seconds
setTimeout(() => {
console.log('About to crash...');
// Flood stdout with data right before crashing
for (let i = 0; i < 10000; i++) {
process.stdout.write(`Pre-crash log line ${i}\n`);
}
process.exit(1); // Hard exit — no cleanup
}, 2000);Run this script while piping its output to a file: node lossy-logs.js > output.log. Then count the lines in output.log. You'll almost certainly find that not all 10,000 lines made it. The exact number you lose depends on your system's pipe buffer size, the timing of the flush, and other factors — but you will lose some. This non-determinism is what makes the bug so hard to reason about. It doesn't fail consistently, which means it often gets chalked up to "weird timing issues" and never properly investigated.
What Console.log Is Actually Doing
Here's something that surprises a lot of developers: console.log is not a direct write to stdout. It's a higher-level abstraction that internally calls process.stdout.write(), but it adds formatting, handles argument serialization, and — importantly — it inherits whatever buffering behavior process.stdout is operating under at that moment.
When stdout is in TTY mode (connected to a terminal), process.stdout.write() is actually synchronous. It blocks the event loop until the data is written. This is why your local development experience feels so reliable. But in non-TTY mode (pipes, files, containers), it becomes asynchronous. The write is handed off to libuv (Node's underlying I/O library, which handles all the cross-platform async stuff), and your code moves on immediately while the actual write happens in the background.
You can check which mode you're in at runtime:
// check-stdout-mode.js
if (process.stdout.isTTY) {
console.log('Running in TTY mode — writes are synchronous');
} else {
console.log('Running in piped/file mode — writes are async and buffered');
console.log('Log data could be lost on abrupt exit!');
}
// Check if stdout is writable and what kind of stream it is
console.log('stdout fd:', process.stdout.fd);
console.log('stdout writable:', process.stdout.writable);
console.log('stdout writableNeedDrain:', process.stdout.writableNeedDrain);The writableNeedDrain property is particularly useful — it tells you whether the stream's internal buffer is full and needs to drain before you can safely write more. If you're logging heavily and this is true, you're already in dangerous territory.
How to Actually Protect Your Logs
The most robust solution I've found is to use a synchronous write when you're in non-TTY mode, or to explicitly drain the stream before exiting. Node.js provides fs.writeSync for truly synchronous file writes, and you can redirect your logging through it:
// safe-logger.js
const fs = require('fs');
function safeLog(message) {
const line = `${new Date().toISOString()} ${message}\n`;
if (process.stdout.isTTY) {
// In TTY mode, console.log is fine
process.stdout.write(line);
} else {
// In piped/file mode, use synchronous write to stdout's file descriptor
// fd 1 is always stdout
try {
fs.writeSync(1, line);
} catch (err) {
// If stdout is closed, try stderr (fd 2) as fallback
try {
fs.writeSync(2, `[STDOUT CLOSED] ${line}`);
} catch (e) {
// Both are closed — nothing we can do
}
}
}
}
// Handle graceful shutdown
function gracefulShutdown(signal) {
safeLog(`Received ${signal}, shutting down gracefully`);
// Give stdout a moment to drain if it's async
if (!process.stdout.isTTY && process.stdout.writableNeedDrain) {
process.stdout.once('drain', () => {
process.exit(0);
});
// Safety timeout — don't wait forever
setTimeout(() => process.exit(0), 3000);
} else {
process.exit(0);
}
}
process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));
process.on('uncaughtException', (err) => {
safeLog(`UNCAUGHT EXCEPTION: ${err.message}\n${err.stack}`);
process.exit(1);
});
// Test it
safeLog('Application started');
safeLog('This log will not be lost');The fs.writeSync(1, ...) call is the key piece here. File descriptor 1 is always stdout by convention on Unix-like systems and Windows. By calling the synchronous version directly, we bypass the stream buffering entirely and force the write to complete before moving on. Yes, this blocks the event loop. For logging, in most cases, that's acceptable. You're paying a small performance cost in exchange for guaranteed delivery.
The Role of Log Aggregators and Docker
The problem gets even more complex in containerized environments. When you run a Node.js app inside Docker, stdout is typically connected to Docker's logging driver. That driver might be writing to a file, forwarding to a remote service like Fluentd or CloudWatch, or doing something else entirely. Each of these adds another buffer, another potential point of loss.
Docker itself uses a JSON file logging driver by default, which buffers writes to disk. Kubernetes adds yet another layer — it captures container stdout, writes it to files on the node, and then a DaemonSet (a process that runs on every node in the cluster) like Fluentbit or Filebeat picks those files up and ships them to your central log storage. Every hop is a potential buffer. Every buffer is a potential point of data loss if something crashes at the wrong moment.
The practical takeaway for containerized deployments is to treat your logging infrastructure as unreliable by default and design accordingly. Use a dedicated logging library like Pino or Winston that gives you explicit control over write strategies. Pino, in particular, was designed with this exact problem in mind — it writes to stdout in a way that minimizes the risk of loss and offers a separate worker-thread transport mechanism to keep logging off the main thread without risking data loss on shutdown.
Signals, Cleanup, and the Exit Trap
One pattern I see constantly in Node.js code is attaching a process.on('exit') handler and putting logging calls inside it. This looks reasonable on the surface — you want to log something when your app shuts down, so you hook into the exit event. The problem is that the exit event only fires synchronous callbacks. Any asynchronous operations you try to start inside that handler — including buffered stream writes — will never complete.
When your process receives a signal like SIGTERM from Kubernetes or PM2 and you haven't handled it explicitly, Node.js defaults to terminating the process immediately. No cleanup, no flushing, no goodbye. If you have a process.on('exit') handler, it runs, but since it's synchronous-only, any pending buffered writes in stdout that were initiated before the exit event fire can still be lost because the event loop shuts down immediately after those synchronous exit handlers complete.
The correct pattern is to handle signals explicitly with process.on('SIGTERM', ...) and process.on('SIGINT', ...), perform your cleanup synchronously or wait for async operations with a timeout, and then call process.exit() yourself. This gives you control over the shutdown sequence and lets you ensure buffers are drained or final log lines are written synchronously before the process dies. It's not glamorous infrastructure work, but in my experience it's the difference between being able to diagnose production incidents in minutes versus hours.
Choosing the Right Tool for the Job
Not every application needs fs.writeSync. The synchronous approach has a real performance cost at very high log volumes. A better long-term solution for most production services is to use a structured logging library that handles these concerns for you.
Pino is my personal recommendation. It offloads actual log writing to a separate worker thread using its transport mechanism, which means your application code never blocks, but the writes happen in a dedicated process that can properly manage its own buffers and handle shutdown cleanly. Winston is more configurable and has a massive ecosystem of transports, but requires more careful configuration to avoid the buffering pitfalls we've discussed. Whatever you choose, make sure you understand its flush behavior and test it explicitly under your production-like environment, not just your local terminal.
The broader lesson here is that the environment your code runs in fundamentally changes how it behaves. Stdout is not a simple pipe. Streams are not transparent. Buffers are everywhere, and they don't care about your debugging needs. Understanding the machinery underneath your console.log calls isn't premature optimization — it's the kind of knowledge that separates applications that are debuggable under pressure from ones that leave you staring at incomplete logs at 2am wondering what actually happened.