Skip to main content
SUNDAY SYSTEMS DIGEST · ONE MEASURED ESSAY EVERY WEEKREAD · BENCHMARK · REFLECTZERO TRACKING PIXELS · FULL RSS & MARKDOWN ARCHIVENEW BENCHMARK: DETERMINISTIC RAFT LEASE QUORUMS
KERN.

Search Archive

Recent & Most Read Essays (8)

Content-Addressed Monorepo Build Caches Without Silent CI Drift

Why timestamps, unsorted directory globs, and leaked host `PATH` variables poison remote build caches—and a 120-line digest verifier that pushed our cache hit rate from 61% to 96.4%.

Henrik Lindqvist
Henrik Lindqvist

Principal Systems Architect · Stockholm

Nov 23, 20258 min read205 words
Content-Addressed Monorepo Build Caches Without Silent CI Drift

A remote build cache is only as trustworthy as its action input Merkle tree. Teams adopt task runners expecting five-minute pull requests to drop to forty seconds, only to discover two weeks later that macOS laptops never hit artifacts compiled on Linux runners—or worse, a stale .d.ts bundle ships to production because an undeclared environment variable changed.

Four Silent Sources of Cache Misses and Poisoning

Auditing 11,400 CI executions across a 90-package TypeScript and Rust repository exposed four recurring culprits:

  1. Locale-dependent file glob ordering: readdir returns inode order on ext4 and APFS-normalized order on macOS unless explicitly sorted by UTF-8 byte code points.
  2. Ambient environment pass-through: Hashing process.env wholesale busts every task whenever GITHUB_RUN_NUMBER increments; omitting env vars lets NODE_ENV drift poison outputs.
  3. Embedded archive mtimes: tar headers embed modification timestamps unless --mtime='UTC 2020-01-01' and --sort=name are enforced.
  4. Absolute diagnostic paths: Compiler source maps emitting /home/runner/work/... instead of workspace-relative ../../packages/... paths.
ts
export function computeTaskDigest(inputs: {relPath: string; blake3: string}[], envAllowlist: Record<string, string>): string {
  const sortedFiles = [...inputs].sort((a, b) => (a.relPath < b.relPath ? -1 : a.relPath > b.relPath ? 1 : 0));
  const sortedEnv = Object.keys(envAllowlist).sort().map((k) => `${k}=${envAllowlist[k]}`);
  return hashPayload(JSON.stringify({ v: 2, files: sortedFiles, env: sortedEnv }));
}

Double-Execution Determinism Gate

Once every night, run a clean build twice in two isolated temporary directories (/tmp/build-a and /tmp/build-b) with TZ=UTC and compare output hashes byte-for-byte. That single nightly check kept our frontend accessibility checks in Top-Layer HTML Dialogs and Tactile Braille Focus Discipline and Wasm AOT artifacts in Anatomy of a Sub-Millisecond WebAssembly Component Cold Start reproducible across every developer machine.

Connected Architecture Notes

Henrik Lindqvist

Henrik Lindqvist

Stockholm · UTC+1

Principal distributed systems architect writing weekly benchmarks on consensus protocols, storage engines, WebAssembly isolates, and tactile web accessibility.

Reader Field Notes

0 verified notes

No published field notes yet on this essay. Share a production observation or benchmark note below.

Contribute a Technical Note

Moderated manually for signal-to-noise. Your email is never stored in public comment properties.

Enter your display name.
Enter a valid email or leave blank.
Please write at least 8 characters.
Please confirm publication consent.

Continue Reading

All essays →

Get next Sunday’s benchmark by email

Full essay in plain HTML, zero tracking pixels, one-click unsubscribe.

Enter a valid email address (for example, name@domain.com).