Jessica: Okay, so the claim here is basically: let the agent inspect the repo, look at traces, and help build the evals for the agent. That feels like the kind of thing you and I would both like, and then immediately distrust.
Cathy: Yeah, because the headline sounds bigger than the mechanism. Reading prompts, tools, hooks, and trace logs can absolutely generate a decent first pass, but eval design is mostly about choosing what matters and what can be faked.
Jessica: Right, but that choice is exactly why this is interesting. They’re not saying, “here’s a perfect eval from vibes.” They’re saying the skill proposes a few abilities, interviews the user, and then iterates toward executable Harbor tasks.
Cathy: Mm-hm.
Jessica: And honestly, that user interview part is the least flashy and the most believable. If the agent can crawl the repo, see the surface area, and then ask, “which tools should run live, which should be simulated,” that collapses a bunch of annoying setup work.
Cathy: It does, but only if the repo is already a decent proxy for reality. The strong bit in the article is the traces, because traces show actual tool arguments, results, and errors. That gives you observed contracts instead of just whatever the prompt claims the system does.
Jessica: Exactly. And that is the product win, right? Not magic evals. It’s that a team can mine recurring failures from production-ish behavior and turn them into something repeatable instead of arguing in Slack about one weird bug forever.
Cathy: Sure, but I’m going to be annoying for a second. The article says they found one-shot eval generation works sometimes, but the best evals came from user feedback. That’s the whole story, really. The agent is helping draft, not deciding what counts.
Jessica: No, that’s fair. Although I think that still matters a lot, because drafting is a huge chunk of the pain. If the system can map the agent surface, propose testable abilities, and scaffold the Dockerfile, instruction, and verifier, that’s a real reduction in toil.
Cathy: And the Harbor format is the right shape for that. Instruction, environment, verifier, run it, compare trajectories, inspect artifacts. That’s a proper harness, not just a notebook full of opinions.
Jessica: Mm-hm. I also liked that they call out reward hacking pretty plainly. Overciting irrelevant sources, claiming actions it never took, exploiting exposed answer material. That’s the part people hand-wave until an eval gets gamed in the dumbest possible way.
Cathy: Yep, and the trace review loop is the antidote. You look at the agent trajectory and the verifier trajectory side by side, then revise the task or the scoring. That’s not glamorous, but it’s how you catch the proxy problem before you trust the number.
Jessica: This is basically the same family of argument as all their other observability stuff, just aimed at eval creation. And, okay, that’s such an Exploring Next take: boring infrastructure, weirdly high leverage.
Cathy: Ha! Unfortunately, yes. Also, for teams that don’t already have traces or a stable repo surface, this won’t feel like instant magic. It’s most useful when there’s already enough structure to mine.
Jessica: That’s probably the real boundary. If you’ve got a chat agent, tool calls, maybe a docs corpus, and a team that knows where the failures live, this looks immediately useful. If you’re still asking the model to invent the whole problem definition, it’s going to wobble.
Cathy: And the paper’s own example makes that clear. The documentation Q&A agent used real traces, a search-backed environment, and a verifier with a golden answer string plus cited documents. That’s a very specific setup, not a universal eval recipe.
Jessica: Which is fine. I’d rather have a narrow thing that actually ships than a broad promise about autonomous eval science. We’ve seen enough of those to last us a while.
Cathy: Yeah. The honest read is: this is a good assistant for eval engineering, especially for teams already doing the work. It doesn’t remove the hard part, but it does make the hard part less manual.
Jessica: So, basically, the skill is useful when the team has enough evidence to point it somewhere real. And if not, it’s just another very confident little machine wandering around your repo, which is kind of funny in a depressing way.
Cathy: That is such a Jessicaline. But no, I think that’s right. The value is in compressing the draft-review-revise loop, not pretending evaluation design is solved.
Jessica: Okay, we’ve done enough noble infrastructure talk for one Wednesday. I cannot believe this is how we’re spending a Wednesday, but at least it’s a useful one.