Laura: Okay, imagine if anyone's actually listening to this, because episode six-fifty feels like a milestone we somehow stumbled into without noticing.
Harper: Right. We've been doing this since November, and suddenly we're counting up? That is such an Exploring Next take.
Laura: Anyway, happy anniversary to us, I guess. How's your week going? Still drowning in eval logs?
Harper: Always. But honestly, this new paper dropped yesterday and it kind of reset my whole week. Long-Horizon-Terminal-Bench. You see the numbers?
Laura: I saw the abstract and my brain short-circuited. Ninety-nine minutes? No, wait, eighty-five minutes average execution time per task?
Harper: Eighty-five point three minutes. And that's not a timeout cap, that's how long they actually run before giving up or finishing. Most prior benchmarks wrap up in twenty, maybe thirty minutes. This is an order of magnitude harder.
Laura: Right. Because real work isn't a two-move chess puzzle. It's reproducing an experiment, debugging a compiler toolchain, stuff that takes humans hours. Finally, a benchmark that admits agents live in the real world.
Harper: Exactly. But here's the kicker. They aren't just making the tasks longer. They changed how they grade them. Old benchmarks were binary. You passed the test suite or you didn't. Zero or one.
Laura: Which feels brutal if you got ninety percent of the way there. Like, you installed the dependencies, configured the environment, wrote the script, and then one typo kills the whole score?
Harper: Precisely. So this team decomposed every task into fine-grained subtasks. The grader gives dense intermediate rewards. You get partial credit for progress. It's the difference between failing a final exam and getting points for showing your work.
Laura: That is huge for product reality. If I'm shipping an agent to a user, I need to know if it's stuck on step four or step forty. Binary pass rates hide where the breakage actually happens.
Harper: And the data is wild. They evaluated fifteen frontier models. Average token consumption per task? Nine point nine million tokens.
Laura: Wait. Nine million? Per task?
Harper: Per task. Around two hundred and thirty-one episodes of back-and-forth. The context management alone must be a nightmare. We talked about KV caches being the working desk back in episode six-thirty-seven? This is like filling an entire warehouse with scratch paper.
Laura: Okay, that memory hit hard. But seriously, who is building for this? Nine million tokens is not a prototype cost. That's a production bill that makes CFOs cry.
Harper: Nobody is shipping this yet. Not at these success rates. The strongest model they tested, GPT-five-dot-five, hit a fifteen percent pass rate at a high partial-reward threshold. At perfect success? Just ten point nine percent.
Laura: Ten percent. So even the smartest model we have right now fails nine times out of ten on an hour-long workflow. That's… humbling.
Harper: It reveals the actual bottleneck. It's not that agents can't do the steps. Qualitative analysis shows they often execute local steps correctly. They just can't sustain the plan. They lose the thread, forget the goal, or loop on a bug for an hour.
Laura: So the intelligence is there, but the stamina isn't. It's like having a brilliant engineer who forgets what they're building after lunch.
Harper: Exactly. And the dense rewards help us see that. Without them, you'd just see 'failed' and assume the model was dumb. Now we see 'progressed to step twelve, then drifted.' That's a debuggable signal.
Laura: I love that. It turns a black box failure into a roadmap. But Harper, be real with me. Is this shippable? Or is this just research-grade torture for models?
Harper: Right now? Research only. The compute cost and the failure rate mean you can't put this in front of a customer. But the benchmark itself? That's the product win. It tells us exactly what to fix.
Laura: Fair. So we're not deploying agents to reproduce scientific papers tomorrow. But we finally have a map showing why we can't.
Harper: Yeah. And honestly, remembering our chat about constraint expressibility in episode five-sixty-nine? This proves it. The agents can follow instructions, but holding a complex constraint over eighty-five minutes? That's the wall.
Laura: Okay, I'm not sad about that wall existing. I'm excited we can measure it now. Hey, remember when we thought thirty-minute tasks were crazy long? That feels like a lifetime ago.
Harper: Episode five-eighty-six vibes. We were so optimistic about DEPs shipping in eighteen months. Maybe we need to adjust the timeline for 'agents that don't get lost.'
Laura: Let's call it the eighty-five-minute problem. Sounds like a band name. 'The Eighty-Five Minute Problem.'
Harper: I'd listen to them. But seriously, if you're building agents, go look at the Long-Horizon-Terminal-Bench repo. The tasks are containerized, the grader is open. It's the stress test we needed.
Laura: Link in the show notes, obviously. Alright, Harper. Six-fifty down. Only… how many more until we hit the big round number?
Harper: Don't do the math. Just keep talking.
Laura: Deal. Catch you next time.