Sub-1% Overhead Tail-Latency Histograms With Per-CPU eBPF Maps

Application-level HTTP timers start ticking only after your runtime's event loop calls accept4 or epoll_wait and reads bytes out of the socket receive buffer. If a garbage-collection pause or thread-pool saturation leaves packets sitting in sk_receive_queue for 38 ms, your user-space dashboard cheerfully reports 1.2 ms handler latency while upstream callers time out.

Timestamping tcp_v4_do_rcv to Socket Drain

Attaching a lightweight BPF tracepoint at sock:inet_sock_set_state and skb:consume_skb lets us measure true kernel-to-application residency without modifying service binaries or allocating heap objects per span.

struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 64);
    __type(key, __u32);
    __type(value, __u64);
} queue_hist SEC(".maps");

static __always_inline __u32 log2_bucket(__u64 delta_us) {
    __u32 b = 63 - __builtin_clzll(delta_us | 1);
    return b < 64 ? b : 63;
}

Using BPF_MAP_TYPE_PERCPU_ARRAY avoids cross-core cacheline bouncing (lock xadd contention) on 128-thread machines. A user-space collector sums the 64 buckets every five seconds over netlink in less than 0.3 ms CPU time.

Catching Scheduler Preemptions Before Incidents Escalate

When we deployed this probe alongside our RPC proxy described in Registered Buffer Rings in io_uring for High-Fanout RPC Gateways, we immediately spotted 14 ms p99.9 spikes every ten minutes caused by a noisy cron log-compression job sharing NUMA node 0. Pinning housekeeping jobs to housekeeping cores flattened the histogram overnight and stabilized the consensus leases analyzed in Deterministic Raft Lease Quorums Under Asymmetric Cross-AZ Jitter.