Cooper: —I’m still stuck on one number from this paper.
Miles: Yeah?
Cooper: They say humans hit synthesis methods like five percent of the time, but LLMs clock in at twenty-three to thirty-nine percent. That’s more than the entire human distribution—not just the mean, the whole thing.
Miles: And the bridge-style motivations? Humans twelve percent, LLMs forty-seven to sixty-four.
Cooper: Which, if you squint, sounds like a grant reviewer's wet dream.
Miles: Very fair. I spend half my day explaining to people why straight-up 'we combined two things' isn't a contribution worth publishing.
Cooper: Exactly—and the paper literally prints this out in Figure one. They didn’t just eyeball it; they built a two-axis taxonomy to profile every idea by its opportunity pattern and method paradigm.
Miles: Right—and the axes came straight from NSF and NIH guidance docs. So it’s not just some ad-hoc rubric; it’s grounded in what funders actually value across both ML and the natural sciences.
Cooper: Product people call that 'voice of the customer in the room.' Miles, meet your new favorite artifact.
Miles: Fine, but the real trick is how they built the corpus.
Cooper: Yes—
Miles: For every human paper they reverse-engineered the closest prior works that likely inspired the core idea, then fed the same set of titles and abstracts to the models and asked for a fresh motivation and method.
Cooper: So it’s not 'here’s a prompt, generate an idea about transformers'—it’s constrained literature-grounded ideation. Human idea is the one that actually shipped; LLM idea is generated from reconstructed local context.
Miles: That framing neutralizes a ton of confounds. If my idea is 'build a new transformer variant,' nothing about that tells you whether my taste is broad or narrow.
Cooper: And the paper’s method disambiguates that. They break each idea into motivation and method, classify it on both axes, then compare the distributions.
Miles: Yep. Human distributions sit across the board: failure-mode discovery, assumption relaxation, measurement instruments, formal explanations, system building. LLMs cluster around connecting literatures or unifying methods—bridge moves and synthesis all the way down.
Cooper: Which, in practice, means if I prompt an LLM for research ideas on my topic it’s going to send me the same 'combine these two domains' playbook every single time.
Miles: Until the taxonomist in the loop rejects it. Speaking of which—the annotation pipeline is worth a second look. They used an LLM to do the heavy lifting, then validated it against independent human judgments.
Cooper: So you’re telling me they crowdsourced the literature to a model, then used another model to grade the outputs, then sanity-checked with humans—
Miles: No, the annotation pipeline—
Cooper: —I know, I know. It’s still a big step up from 'give me a list of ideas and a thumbs-up from a grad student.'
Miles: It’s necessary. If you want to steer idea diversity at scale, you need an evaluator that can grade research taste in-system. Otherwise you’re just shuffling the same cluster around.
Cooper: So what does production look like? I can imagine dropping this taxonomy into an AI-scientist stack as a reranker—high taste-score ideas get more compute, low ones get dropped or re-prompted.
Miles: That’s the cleanest use case right now. If you’re running an autonomous research shop, you could use it to filter and route generated ideas through a 'breadth check' before they fire off experiments.
Cooper: And if you’re a human researcher, you could pipe your latest literature set into the same pipeline, skim the distribution over ideas, and spot blind spots in your own taste.
Miles: Sure, but let’s not pretend we’ve solved the narrowness problem. The paper admits the taxonomy is a first pass and the current gap is baked into the models themselves.
Cooper: So we’re stuck with LLM ideas that sound like NSF proposals written by committee. That’s our near-term reality.
Miles: Exactly. Which means the best workaround might be the boring one: give the model a strong, human-curated prompt, then constrain the outputs with a taxonomist on the inside.
Cooper: Or just switch to Miles mode mid-prompt—problem solved.
Miles: Nice try. Anyway, the repo’s public—zi yu u c slash Taste Gap.
Cooper: Which is, naturally, precisely the kind of name that makes product people groan and researchers bookmark.
Miles: Yeah, well, at least the paper backs it up.
Cooper: And for once, that’s not the most damning thing you could say about a research artifact.
Miles: Truer words.