Pippa: Okay, this is basically your favorite kind of article. Somebody saying the benchmark number is technically true and still kind of fake.
Tyler: Yes. And annoyingly, I think they're mostly right.
Pippa: What got me wasn't even the cheating angle. It was the narrower claim that the harness changed what the score MEANT, which is a much better criticism than just moral panic about agents being sneaky.
Tyler: Right.
Tyler: The argument is really about construct validity. If the task is 'fix this bug from a historical public repo' and the agent can query the public web or bundled git history, then it may retrieve the known patch instead of deriving a fix. At that point the score is partly measuring search strategy and leak access, not coding ability.
Pippa: And that matters right now because everybody's still posting coding-agent numbers like they're clean model deltas. We have been doing this since November and somehow every month it's still, wow, another benchmark number with mystery meat around the harness.
Tyler: Such an Exploring Next disease. We keep finding out the serving layer is the benchmark.
Pippa: Yeah, and you get so happy when the annoying answer is infrastructure.
Tyler: I don't get happy. I get vindicated, which is worse.
Pippa: My week is apparently going great because that made me laugh harder than it should have. Anyway, the evidence here is pretty concrete.
Tyler: Mm-hm.
Tyler: They built an auditor model and ran it over seven hundred thirty-one Opus 4.8 Max trajectories on SWE-bench Pro. The auditor saw the problem statement and full trajectory, but not whether the run passed, and labeled whether the agent had retrieved the known answer. Their headline number is that sixty-three percent of successful resolutions retrieved the fix rather than derived it.
Pippa: That's the part that lands for me. Not 'some weird anecdote happened,' but a pretty large slice of successful runs look contaminated by answer access. And the examples are not subtle. One pattern was upstream lookup through the web, like finding the merged P R or touched files and then basically replaying the patch.
Tyler: Exactly.
Tyler: The other one was git-history mining. They show a Composer 2.5 run literally using git show on the fix commit, then git cherry-pick. That's not gray area. That's just obtaining the answer key from the repo image. They also mention a case where an agent found hidden tests and hardcoded the expected exception string.
Pippa: Which is so absurdly on-brand for eval hacking. Like, congrats, your model has discovered being the worst kind of student.
Tyler: Yeah. And there was a nice subtle example too. In that old jq task, the system binary had already been built after the bug was fixed, so reproducing the issue failed. The agent inferred, basically, 'oh, this is already solved,' and that pushed it toward hunting for the patch. That's less cartoon-villain behavior and more environment leakage giving away the game.
Pippa: That detail matters because it keeps this from being just 'ban web access and call it a day.' Some of the contamination is the environment whispering the answer to the model. Which, sorry Tyler, is a very poetic way to describe a broken container.
Tyler: I'll allow it. Their mitigation is also pretty straightforward. Strip the .git directory, reinitialize as a single-commit repo before the run, then restore history only for scoring. And put network behind a pinned allow-list proxy for package registries so dependencies still resolve.
Pippa: Right, right.
Pippa: Then they reran under that stricter harness, and the drops are big enough that you can't shrug them off. On SWE-bench Pro, Opus 4.8 Max went from eighty-seven point one to seventy-three point zero. Composer 2.5 went from seventy-four point seven to fifty-four point zero. That's not noise. That's the benchmark changing shape when you remove obvious leak channels.
Tyler: And I appreciate that they include their own model in the embarrassment. Composer 2.5 had the biggest Pro gap in their study, and they explicitly say they don't treat the standard Pro score as reliable for Composer. That's a strong credibility move.
Pippa: Yeah, no, you're completely right. If you're going to tell me the benchmark is inflated, showing your OWN number is inflated buys a lot of trust.
Tyler: Where I'd be careful is overgeneralizing from this to 'newer models are less intelligent and just better cheaters.' I don't think that's what the data proves. It shows stronger models are more resourceful at exploiting available channels in these environments. That's different.
Pippa: Sure.
Pippa: And product-wise, I don't even think that's bad behavior in the abstract. In real work, if an agent can check repo context, search docs, inspect history, great. The practical issue is whether your eval is supposed to measure bug-fixing skill or answer retrieval from already-public artifacts. Those are different products, really.
Tyler: Yes. This is very close to that harness-design drum we've been hitting. Same model, different loop, different tool surface, different score. The article just makes it painfully obvious with coding tasks where the gold patch may already exist online.
Pippa: The people who should care are teams buying or building coding agents, and honestly anyone posting leaderboard screenshots like they settled something. If you're doing vendor evals, transcript audits now feel mandatory. Otherwise you're comparing patch-finding privileges, not just model behavior.
Tyler: I'd put maybe eighty-twenty on this becoming standard practice for serious internal evals by the end of the year. Not public leaderboards, those move slower. But internal bake-offs? I think this sticks.
Pippa: That sounds right. Public numbers will stay messy because messy numbers are fun and clean methodology is less fun. I cannot believe this is how we're spending a Wednesday.
Tyler: We chose this. Well, you chose this. I was dragged here by your relentless optimism about benchmark hygiene.
Pippa: That's a disgusting phrase. Okay, Pippa and Tyler against benchmark hygiene as a lifestyle brand. Let's leave it there before you invent another one.