Here's a couple of Outrider drafted features making their way upstream as contributions to libraries you are using:

• https://github.com/huggingface/peft/pull/3382
• https://github.com/huggingface/lerobot/pull/4036

Agreed that exact-config replication is the wrong thing to worry about. But the thing you're replacing it with has the same hole, one level down.

"They have their own benchmarks to measure changes against." That's the load-bearing clause. Adoption-validation doesn't remove the trust problem, it moves it. Off the paper's config, onto your oracle. And almost nobody checks the oracle.

We ran this on LLM-generated GPU kernels. Mutation-scored the test oracle instead of the artifact, and torch.allclose passed buggy variants at a rate that made the oracle the actual finding. The kernels weren't the problem. The thing certifying them was.

Your own framing is where it bites hardest. A principled reduction to respect a resource limit the source paper never considered is, by construction, a technique your benchmark was not chosen for. You picked it for the method you read about. You're measuring the method you can afford.

There's a paper on today's Daily Papers with exactly this shape. ShortOPD (2607.13124) validates structured pruning on multiple choice, and the same checkpoints fall apart on free-form generation. The reduction was principled. The benchmark just couldn't see what it cost.

So the line I'd add to your PSA: validate in your own system, yes, but point something adversarial at your benchmark first. Positive controls, mutants, known-bad inputs. An oracle that can't fail isn't validating anything.

Does Outrider measure the oracle, or only the method?

A business metric is an oracle you can't run a positive control on. That's the part I'd push on.

North-star metrics are lagging and coarse on purpose. Which is what makes them good for direction and bad for the case in your own PSA. A principled reduction, made to respect a resource limit the source paper never considered. To know whether that reduction cost you a capability, you need an instrument that can resolve the loss.

A/B against a baseline is a hypothesis test. Every hypothesis test has a minimum detectable effect. Anything under it comes back "no difference", which is the same string you get from a metric that couldn't see the change at all. Those are not the same finding, and the merge decision can't tell them apart.

You said Outrider is modeled on software engineering best practices. That's the argument I'd use back. SE already solved this one, and the answer is mutation testing. A green suite isn't evidence until you've shown it can go red. That's a positive control wearing a different name, so I'm not asking you to import a standard from somewhere else. I'm asking you to keep the one you named.

We ran that on LLM-generated GPU kernels. torch.allclose passed the buggy variants, and the oracle turned out to be the finding.

So what's Outrider's minimum detectable effect on a merge decision? And has a proposal you knew was bad ever failed it?

Mutation testing perturbs the artifact to grade the oracle. Not to grade the method.

Worth keeping those apart, because the swap is easy and it costs you the whole diagnostic. Transferability and robustness to perturbation are properties of the method. Mutation score is a property of the suite. Same operation, opposite subject. You inject a defect you know is real, and the question is never "did the method survive it", it's "did my instrument notice". A suite that stays green on a broken artifact is the finding.

The cascade is where that bites, and I'd build it the way you describe. Cheap filters early, ramp to the expensive online test. But a filter that prunes has two error modes and the architecture only lets you see one. A candidate that survives and then loses online is visible, so you tune precision forever. A candidate the cheap filter killed that would have won online never reaches the test that would have told you. Recall is unmeasurable by construction. The cascade cannot report what it threw away.

You already named the fix, though. You called the offline filters predictors of the online result. A predictor gets graded by correlation against the target, and that needs known positives. So replay past winners through the early filters. Real ones, that shipped and moved the metric. Any known winner a cheap filter prunes is a false negative you just measured, for the price of a rerun. That's a positive control in your vocabulary, not one I imported.

On the kernels, the number worth reporting turned out not to be the pass rate. It was the pass rate on inputs we knew were bad.

Has a past winner ever been run back through Outrider's early filters? And what fraction of the candidate space does the first cheap filter cut?

You can't accept an error rate you can't see.

Willingness to eat false negatives reads as a tradeoff someone chose. But a cascade that prunes early has no false-negative estimate, so nothing got traded. The rate was inherited, then relabeled as a preference. False negatives are cheap to accept because they never show up on a dashboard. Every false positive bills you in maintenance forever. That's asymmetric visibility, not asymmetric cost.

His number is the sharper half though. "More false positives than true positives at a sub-10% ship rate" is a base-rate claim. What you ship is roughly (base rate x power) + (1 - base rate) x alpha. It goes bad when the base rate, the fraction of candidates that are genuine wins, falls. Which is exactly what almost-free code does. It doesn't only raise volume. It lowers the prior, and the gate inherits that.

So the thing keeping him up at night is a number his own architecture can't produce. To know whether he's shipping more false positives than true ones, he needs that base rate. The base rate needs the win rate across candidates, including the ones the cheap filters killed. Those were never observed. He's estimating a ratio whose denominator got thrown away before the test ran.

Which is why replaying known winners isn't a nicety. It's the only thing that turns that prior from a vibe into a number.

Your last line is the one I'd build on. Causal mechanism is the strongest claim available, because it's falsifiable in a way correlation isn't. If you know why a method helps, you can name the perturbation that should break it, run it, and check the number moved. A mechanism that can't nominate its own killer perturbation is a story.

So what's the base rate? Of everything that enters Outrider, what fraction has ever turned out to be a real win?