What Even Is eBPF?
The Linux kernel is the beating heart of almost every server, cloud instance, and container running today. It manages memory, handles system calls, talks to hardware, and basically keeps everything from descending into chaos. For decades, if you wanted to change how the kernel behaved — add new monitoring, tweak networking, intercept system calls — you had two options: write a kernel module (risky, complex, one bad pointer dereference and your whole system crashes) or patch the kernel itself (good luck with that in production). eBPF changed the game entirely.
eBPF stands for extended Berkeley Packet Filter. Don't let the name fool you — the original BPF was indeed about filtering network packets, but eBPF has evolved so far beyond that origin that the name is almost comically misleading. Think of eBPF as a tiny, sandboxed virtual machine that lives inside the Linux kernel. You write small programs, the kernel verifies they're safe, and then it runs them at specific hook points — network events, system calls, function entries, hardware performance counters — without you ever needing to touch or recompile the kernel itself.
The analogy I keep coming back to is JavaScript in a browser. Before JavaScript, if you wanted a webpage to do something dynamic, you basically had to ship new browser code. JavaScript gave developers a safe, sandboxed way to run custom logic inside the browser without breaking it. eBPF does the same thing for the Linux kernel. You get incredible power and visibility without the catastrophic risk of kernel crashes or security holes from poorly written kernel modules.
The Verifier: Your Safety Net
Here's what makes eBPF genuinely remarkable rather than just another way to shoot yourself in the foot. Before any eBPF program runs, the kernel's verifier — a piece of static analysis code — examines your program completely. It checks that there are no infinite loops, that memory accesses are safe, that you're not going to trash kernel memory. If your program fails verification, it simply doesn't load. The kernel rejects it cleanly.
I've found this verification step is actually one of the things that makes eBPF easier to trust in production than kernel modules. A buggy kernel module can corrupt memory, trigger a kernel panic, or silently cause data corruption that you won't notice until much later. eBPF programs that slip past the verifier — which is exceedingly rare — run in an isolated environment where the damage they can do is heavily constrained. The verifier isn't perfect, and researchers have found bypasses over the years, but it represents a fundamentally safer programming model than raw kernel modules.
The verifier also enforces that eBPF programs are bounded — they must eventually terminate. You can't write an eBPF program with an infinite loop (at least not in most contexts). The kernel won't allow it. This constraint forces you to write tight, efficient programs, which turns out to be exactly what you want when you're running code on every single network packet or every file open call across your entire system.
Tracing System Calls With eBPF
Let me show you something concrete. One of the most common uses of eBPF is tracing system calls — the requests that user-space programs make to the kernel, things like opening files, reading network data, or creating new processes. The tool bpftrace gives you a high-level scripting language that compiles down to eBPF programs automatically.
Here's a simple one-liner that traces every time any process on your system opens a file:
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s opened %s\n", comm, str(args->filename)); }'That's it. One line. When you run this, you'll immediately start seeing output like nginx opened /etc/nginx/nginx.conf or python3 opened /usr/lib/python3.10/abc.py. You're watching the kernel in real time, with zero performance overhead from logging to disk, zero changes to the programs being traced, and zero kernel modifications. The tracepoint is a predefined hook point in the kernel — like a designated fishing spot — and your eBPF program is the hook that fires whenever the kernel swims past it.
Now let's look at something slightly more involved. Suppose you want to track which processes are making DNS queries — useful for debugging microservices that seem to be talking to unexpected endpoints. Here's a bpftrace script that hooks into the connect system call and logs connections to port 53 (the DNS port):
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_connect
/((struct sockaddr_in *)args->uservaddr)->sin_port == 13568/
{
printf("PID %d (%s) is making a DNS query\n", pid, comm);
}'The 13568 is port 53 in network byte order (big-endian). Slightly awkward, yes, but this tiny script gives you a real-time feed of every process attempting a DNS lookup on your system. No tcpdump, no packet capture overhead, no need to instrument your application code. The kernel itself is your observability platform.
Profiling Performance With CPU Flame Graphs
Performance profiling is where eBPF really makes experienced engineers do a double-take. Traditional profilers are intrusive — they modify the program being profiled, add instrumentation hooks, or require you to recompile with debug flags. This changes the behavior of what you're measuring, which is a fundamental problem in physics (the observer effect) and turns out to be a very real problem in software performance too.
eBPF-based profiling uses a technique called sampling — the kernel periodically interrupts the CPU at regular intervals, records what instruction was executing and the full call stack, and your eBPF program aggregates these samples. The overhead is typically under 1%. I've run this in production on heavily loaded database servers without any measurable impact on query latency. That's not something I'd say lightly — production profiling used to be something you'd do with extreme caution, typically during off-peak hours.
The perf tool combined with eBPF, or Brendan Gregg's BCC toolkit, gives you flame graphs — visualizations where the width of each block represents how much CPU time a particular function consumed. Here's how you'd generate one using bcc's profile tool:
# Sample CPU stacks at 99Hz for 30 seconds across all processes
sudo /usr/share/bcc/tools/profile -F 99 30 > out.profile
# Then use flamegraph.pl from Brendan Gregg's FlameGraph repo
./FlameGraph/stackcollapse.pl out.profile | ./FlameGraph/flamegraph.pl > flamegraph.svgOpen that SVG in a browser and you have an interactive, clickable visualization of exactly where your system spent its CPU time over those 30 seconds. Every function call, stacked up, across every process on the system simultaneously. This is genuinely magical the first time you see it — suddenly the black box opens up and you can see that your "slow application" is spending 40% of its time in a JSON deserialization library you didn't even know it was using.
Securing Your System With eBPF
Security is where eBPF is seeing some of the most exciting production adoption. Tools like Falco, Cilium, and Tetragon use eBPF to implement runtime security — catching bad behavior as it happens rather than trying to prevent every possible attack in advance.
The idea is straightforward. Your application should behave predictably. A web server should open configuration files and network sockets. It should not be executing shell commands, writing to /etc/passwd, or establishing outbound connections to random IP addresses in the middle of the night. eBPF lets you define what "normal" looks like and alert — or even block — anything that deviates from it.
Here's a simplified Python example using the bcc library to write an eBPF program in C that gets loaded from Python. This one detects any attempt to execute bash or sh — a common indicator of a successful code injection attack:
from bcc import BPF
program = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>
int detect_shell_exec(struct pt_regs *ctx) {
char comm[TASK_COMM_LEN];
bpf_get_current_comm(&comm, sizeof(comm));
// Check if the executed binary is bash or sh
if (comm[0] == 'b' && comm[1] == 'a' && comm[2] == 's' && comm[3] == 'h') {
bpf_trace_printk("ALERT: bash spawned by PID %d\\n", bpf_get_current_pid_tgid() >> 32);
}
if (comm[0] == 's' && comm[1] == 'h' && comm[2] == '\\0') {
bpf_trace_printk("ALERT: sh spawned by PID %d\\n", bpf_get_current_pid_tgid() >> 32);
}
return 0;
}
"""
b = BPF(text=program)
b.attach_kprobe(event=b.get_syscall_fnname("execve"), fn_name="detect_shell_exec")
print("Monitoring for shell executions... Ctrl+C to stop")
b.trace_print()This program attaches to the execve system call — the one that actually executes new programs — and checks if what's being executed is a shell. In a real security tool, you'd extend this to check the parent process, the container it's running in, network connections that preceded it, and much more. But even this simple version would catch a wide class of web shell attacks where an attacker tricks a web application into running shell commands.
The eBPF Ecosystem Today
The tooling around eBPF has matured enormously in the last three or four years. You have libbpf for writing low-level C programs, bcc for Python-based development, bpftrace for one-off investigative queries, and higher-level frameworks like cilium/ebpf in Go for production-grade programs. Rust has aya, which has quickly become my personal favorite for writing eBPF programs because Rust's type system catches the kinds of mistakes that are genuinely dangerous in this environment.
Cilium deserves special mention because it's probably the most ambitious eBPF project in production today. It replaces traditional iptables-based Kubernetes networking with eBPF programs that handle packet routing, load balancing, and network policy enforcement directly in the kernel. The performance improvements are substantial — some benchmarks show 2-3x improvement in network throughput compared to iptables at scale, and the reduction in latency is measurable in real workloads. Major cloud providers and companies like Adobe, Capital One, and Datadog run Cilium in production at scale.
The observability story has also been transformed. Datadog, New Relic, and various other APM (Application Performance Management) vendors have moved their agents to eBPF-based implementations. Instead of requiring you to install language-specific agents and instrument your code, they can observe your applications transparently — seeing HTTP requests, database queries, and service-to-service calls without any changes to your application code. For polyglot environments running Go, Java, Python, and Node.js all together, this is a massive operational simplification.
Getting Started Without Going Crazy
If you want to start experimenting, the path of least resistance is bpftrace. Install it on any modern Linux system (you'll need kernel 4.9 or higher, and ideally 5.x for the best experience) and start with the one-liners. The bpftrace GitHub repository has an excellent collection of example scripts covering everything from file system latency histograms to TCP connection tracking.
From there, bcc with Python is a comfortable next step. The learning curve steepens here because you're writing the eBPF programs in C and the user-space orchestration in Python, but the bcc examples directory is genuinely one of the best collections of practical systems programming examples I've ever seen. Each tool has a standalone .py file and a corresponding .8 man page explaining what it does and why. Reading through those taught me more about how Linux actually works than any book I've read.
For production use, I'd point you toward libbpf with C or aya with Rust. These give you CO-RE — Compile Once, Run Everywhere — which solves the portability problem that plagued early eBPF development, where you had to compile your programs on the exact kernel version they'd run on. CO-RE uses BTF (BPF Type Format — essentially debug information embedded in the kernel) to make your programs adaptable to different kernel versions automatically. It's the difference between shipping a product and shipping a science experiment.
eBPF has genuinely changed what's possible in systems programming. The ceiling on observability, performance analysis, and security enforcement has been raised dramatically, and the floor — the minimum skill required to get useful results — has dropped just as dramatically. The kernel is no longer a black box you prod from the outside. With eBPF, it's a collaborator that will tell you everything that's happening inside, in real time, safely, and without requiring you to be a kernel developer to make it happen.