Deploying Safely with eBPF: GitHub's Approach to Preventing Circular Dependencies

Introduction

Every organization that relies on its own platform for hosting code faces a subtle risk: circular dependencies during deployment. At GitHub, we host our source code on github.com, which means if the site goes down, we cannot access the very code needed to fix it. This circular dependency is more insidious than it first appears—deployment scripts may inadvertently create additional loops by reaching out to internal services or downloading assets from GitHub itself. To address this, we turned to eBPF (extended Berkeley Packet Filter) to selectively monitor and block problematic calls. This guide walks you through the same approach, adapted for any environment where deployment safety is critical.

What You Need

• A Linux system (kernel 4.15+ for core eBPF features; 5.4+ recommended for programmable eBPF)
• BCC (BPF Compiler Collection) or libbpf development tools
• Basic familiarity with C or Python (for writing eBPF programs)
• Access to deploy scripts or service code you want to guard
• Root or sudo privileges for loading eBPF programs
• A test environment mirroring production (highly recommended)

Step-by-Step Guide

Step 1: Identify Circular Dependency Types in Your Environment

Before writing any code, understand the forms a circular dependency can take. Three common categories emerged from our analysis:

• Direct dependency: Your deployment script tries to fetch a tool or binary from a service (e.g., GitHub) that is currently unavailable due to an outage.
• Hidden dependency: A tool already on disk checks for an update online before running. If the update server is unreachable, the tool may hang or fail.
• Transient dependency: Your script calls an internal API (like a migration service), which in turn reaches out to an external resource. The failure propagates back.

Map out all network calls your deployment script makes—directly or indirectly. Focus on calls to your own platform (if it's hosted on the same infrastructure) and critical external services. For GitHub, the key risk was any call back to github.com during an outage.

Step 2: Set Up Your eBPF Development Environment

eBPF allows you to run sandboxed programs inside the Linux kernel, intercepting system calls and network operations. You need the following:

1. Install BCC: sudo apt-get install bpfcc-tools (Ubuntu) or equivalent for your distro.
2. Verify kernel support: cat /boot/config-$(uname -r) | grep -i bpf. Look for CONFIG_BPF=y and related options.
3. Optionally, install bpftrace for quick tracing or libbpf for more advanced C programs.
4. Create a dedicated user or script wrapper to test eBPF injection—don't run untested code in production.

Step 3: Write an eBPF Program to Monitor Network Calls

Our goal is to block or log any outbound connection to a specific target (e.g., github.com or an internal service) during deployment. We'll use a program that hooks the connect syscall and inspects the destination IP address.

Here's a simplified example in Python using BCC:

from bcc import BPF

bpf_text = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>

int block_connect(struct pt_regs *ctx, struct sockaddr_in *addr)
{
    __u32 dest = addr->sin_addr.s_addr;
    // Replace with the network byte order IP of github.com (e.g., 192.30.253.113)
    __u32 blocked_ip = 0x71FD1EC0;  // 192.30.253.113 in little-endian
    if (dest == blocked_ip) {
        bpf_trace_printk("Blocked connection to GitHub\\n");
        return 0;  // allow? In practice you'd return -EPERM to block
    }
    return 0;
}
"""

b = BPF(text=bpf_text)
b.attach_kprobe(event="__sys_connect", fn_name="block_connect")
b.trace_print()

Important: In a real deployment, you'd return a negative error code (like -EPERM) to actually block the call. The above only logs. Also, you need to convert hostnames to IPs at load time (or use a resolver map). For production, leverage eBPF maps to dynamically update blocked targets.

Step 4: Attach the eBPF Program to Your Deployment Script

Once the program is tested, you can load it before running your deploy script. For example:

#!/bin/bash
# Load eBPF blocker
sudo python3 block_github.py &
BPF_PID=$!
sleep 2  # ensure program attached

# Run your deploy script
./deploy.sh

# Clean up
kill $BPF_PID

Alternatively, use a long-running eBPF program that automatically applies to all processes, but beware of affecting other operations. Better to scope it to the specific deployment process using cgroups or PID filtering.

Step 5: Test and Review Logs

Run your deployment in a staging environment. Verify that:

• The eBPF program correctly identifies and blocks (or warns about) unwanted connections.
• The deployment script handles the blocked call gracefully (e.g., by using a cached version or retrying safely).
• No false positives occur—legitimate traffic to non-blocked services should remain unaffected.
• Performance impact is minimal (eBPF is designed for efficiency; test with your workload).

Use bpftool prog show and bpftool map dump to inspect state. Check kernel trace pipes for log messages.

Tips for Success

• Start small: Implement eBPF monitoring for a single type of dependency (e.g., direct downloads) before tackling hidden and transient ones.
• Use BCC for rapid prototyping: Its Python front-end makes writing eBPF programs easier than raw C.
• Leverage maps: Store blocked IPs or hostnames in BPF maps so you can update them without recompiling the program.
• Test for circular dependency propagation: Simulate an outage of your primary service (like GitHub) and ensure the deploy script still completes using local assets.
• Monitor performance: eBPF overhead is low, but complex filters can add latency. Profile with tools like perf or BCC's funclatency.
• Document your dependency graph: Revisit it whenever you update deploy scripts or add new tools. Automated eBPF checks are only as good as the known blocked patterns.
• Consider fallback mechanisms: If a blocked call fails, your script should fail gracefully—e.g., fall back to a local artifact repository or a previously downloaded binary.

With these steps, you can replicate GitHub's approach and protect your deployments from the very circular dependencies that could bring down your entire system. eBPF gives you fine-grained control without modifying your application code—a powerful addition to any operations toolkit.