Justy: Okay, LangChain just dropped a way to judge agent traces a hundred times cheaper using Fireworks.
Cody: A hundred times…
Justy: And it matches frontier models on this 'perceived error' thing.
Cody: Perceived error?
Justy: User thinks the AI messed up. Corrected it, rejected the answer, asked again, or the AI just admitted it was wrong.
Cody: Right. So it’s not objective correctness. It’s 'did the human notice something off.'
Justy: I had the worst flight yesterday—three-hour delay, then they lost my bag. Anyway —
Cody: That explains the three coffees.
Justy: This is why fleet traces. They trained it on their own Docs Q&A agent and Fleet.
Cody: Hm. Two very different use cases, though. Docs is technical, Fleet is no-code everything.
Justy: Exactly. That’s the generality test.
Cody: Yeah. And they only used human and AI messages — stripped out all the tool calls.
Justy: They think the signal’s in the conversation, not the plumbing.
Cody: I mean, sure, a lot of it is. But tool calls can be where things go sideways. A failed database lookup might like a perceived error downstream.
Justy: Fair. They even say that’s a lever they’ll pull later.
Cody: But here’s the thing — they’re claiming frontier parity.
Justy: Mm-hm.
Cody: Against frontier model? On what eval? A hundred examples per dataset isn’t a lot to prove generality.
Justy: Okay, but from a product angle, who CARES about perfect parity if it’s a hundred times cheaper?
Cody: There it is. The classic Justy move: 'good enough and we can ship it.'
Justy: I’m saying you can blanket-judge every trace now. Before, you sampled. Now, you can just run this thing on everything.
Cody: Right. And the dataset prep is almost suspiciously simple — no trimming, no fancy prompting.
Justy: That’s the point. It’s supposed to be plug-and-play.
Cody: I’ll believe the cost when I see the exact numbers, but fine, the idea holds water.
Justy: So teams with agents in production can finally see where users are actually hitting errors.
Cody: Or where they they’re hitting errors. Still useful.
Justy: Anyway. If you want in on the beta, they’ve got a sign-up.
Cody: Of course they do.
Justy: I’m grabbing one of your beers and calling this an Exploring Next win.
Cody: That is such an Exploring Next take.