Pacing LSM-Tree Compaction Debt on High-Queue NVMe Drives

Flash controllers lie politely until their internal garbage collector runs out of erased blocks. On paper, a PCIe Gen5 enterprise drive sustains 14 GB/s sequential reads and 7.2 GB/s sequential writes. Put both workloads on the same namespace simultaneously—say, 650 MB/s of Level-2 to Level-3 LSM compaction alongside 45,000 random 4 KB point reads—and 4 KB read p99 jumps from 95 µs to 4.6 ms.

Read/Write Interference Inside NAND Die Planes

A program operation on modern TLC NAND holds a die plane busy roughly 8x to 12x longer than a page sense operation. When a storage engine flushes 64 MB SSTables in unpaced bursts, the NVMe submission queues fill with large write BIOs that sit ahead of foreground point lookups waiting on the same flash channel.

type CompactionPacer struct {
    targetReadP99Micros int64
    currentRateBytesSec atomic.Int64
    minRateBytesSec     int64
}

func (p *CompactionPacer) Adjust(observedReadP99Micros int64, l0Files int) int64 {
    rate := p.currentRateBytesSec.Load()
    if l0Files >= 12 {
        // Emergency L0 backpressure ceiling overrides gentle pacing
        return max(rate*2, 512<<20)
    }
    if observedReadP99Micros > p.targetReadP99Micros {
        next := max(p.minRateBytesSec, (rate*80)/100)
        p.currentRateBytesSec.Store(next)
        return next
    }
    next := rate + (16 << 20)
    p.currentRateBytesSec.Store(next)
    return next
}

Splitting Write Chunks to 128 KB Boundaries

Beyond byte-rate pacing, issuing O_DIRECT compaction writes in 128 KB slices rather than 2 MB buffers allows the kernel kyber I/O scheduler to interleave urgent read requests between flash program units. In our production time-series cluster, slicing writes plus feedback pacing trimmed foreground p99.9 latency by 68% without increasing L0 file accumulation.

Cross-Checking Consensus and Local Embedded Engines

Storage latency spikes rarely stay inside the storage engine. As documented in Deterministic Raft Lease Quorums Under Asymmetric Cross-AZ Jitter, a 120 ms fsync stall on a Raft WAL device can cascade into unnecessary leader elections unless WAL writes sit on an isolated NVMe namespace—a lesson that applies equally to single-node setups in Predictable SQLite WAL Checkpoints in Read-Heavy Edge Services.