How benchmark noise hides GPU regressions
A single latency number is a coin flip with extra steps. Here's why Kernalin returns "credibly faster / equivalent / regressed" instead of "faster / slower."
You changed a kernel, ran it once before and once after, and it came back 3% faster. Ship it?
Almost certainly not — because you have no idea whether 3% is a real improvement or the width of your measurement noise. GPU timing is noisy: clock boosting, thermal throttling, L2 residency, launch overhead, and neighboring work all move the number. A single measurement tells you almost nothing.
The “1% gate” is theater
Many CI setups gate on a fixed threshold: “fail if more than 1% slower.” This feels rigorous and is mostly theater. If your run-to-run variance is 4%, a 1% gate will pass regressions and fail improvements at random. You’ve built a coin flip with a config file.
Distributions, not points
The honest unit of a benchmark is a distribution. Run the kernel many times, keep the samples, and look at the shape. Now you can ask real questions: where’s the median, how wide is the spread, and — crucially — how confident are we that the candidate’s median is actually below the baseline’s?
Bayesian ROPE: a verdict, not a vibe
Kernalin frames the decision as a region of practical equivalence (ROPE) over the ratio of median latencies, estimated with a bootstrap posterior. The output is one of three honest verdicts:
- Credibly faster — the posterior mass is convincingly on the speedup side of the noise floor.
- Equivalent — the change is within the band where the hardware can’t tell the difference.
- Regressed — convincingly slower.
That’s a defensible statement you can gate a merge on, and it accounts for your specific GPU’s noise floor instead of a made-up global threshold.
Your benchmark shouldn’t be able to lie to you. The fix is to stop asking it a yes/no question and start asking it a probability.
Why it matters for regressions
Silent regressions are the expensive ones — the dependency bump or refactor that shaves 8% off a hot kernel and hides inside noise for a month. A distribution-aware, noise-floor-calibrated verdict catches the regression the day it lands, in the pull request, before it reaches production.
This is sample content published to illustrate Kernalin’s engineering perspective.