"Your code is fast - if you're lucky": why your micro-benchmarks lie

In plain terms

A developer has shown that a piece of code can be 30% faster or slower depending on parameters unrelated to the algorithm: symbol order at link-time, environment variable size, memory alignment. In other words, a good part of what we measure in microseconds is down to luck.

The story

The author, on tiki.li/blog/lucky_code, revisits a well-known thread among compilers (Emery Berger, Stabilizer, ASPLOS 2013) and brings it up to date: they execute the same binary, without modifying it, and vary peripheral parameters (build directory name, object file link order, $PATH size). The runtimes move significantly, up to more than 30% on micro-benchmarks. The cause: the alignment of instructions in the L1i/uop caches, the x86 decoder, and branch prediction, which are sensitive to the virtual address of the code.

The point that should disturb everyone: in real life, we compare a PR before/after, note a gain of 8%, and merge. How much of these gains are real signal? How much is alignment noise?

Under the hood

• Stabilizer (Berger 2013) randomizes stack, heap, and code allocation on each run - measurements become distributions, not points.
• Hyperfine + -warmup + -runs 50 does NOT solve the problem: it averages, but the alignment remains the same on each run of the same binary.
• The good practice: recompile with random offsets (Stabilizer, or manual padding) OR execute with -randomize-address and report a 95% IC, not an average.

So what

Three consequences for the craft. One: any performance gain under 10% on a single micro-benchmark is probably noise. Reject it, or redo it under randomization. Two: CI performance regressions (like Codspeed) must adopt randomization, otherwise we will let real regressions pass and panic over noise. Three: compiler and runtime marketing benchmarks (Bun vs Node, Zig vs C…) should be read with this filter. Performance, at this level, is as much about build engineering as it is about the algorithm.