Pippa: Ugh, another article telling you to just buy more GPUs. I mean, come on, the title alone is a joke.
Tyler: Yeah, and the headline claim is still 'scaling LLMs isn't about more GPUs, it's about removing wasted work.' That’s the part I’m skeptical about.
Pippa: Okay, listen—it’s not wrong. The actual insight is measurement.
Tyler: You say that now, but they list twelve ways and half of them sound like cargo-cult ops mantras.
Pippa: Alright, but they start with the right metrics—TTFT, p-ninety-fives, p-ninety-nines, queue time, cache hit rate. That’s where teams always trip up—misdiagnosing the bottleneck.
Tyler: Sure, that part’s solid. Queue time and TTFT are real pain points. But once they leave measurement… they start recommending 'route everything to the smallest model' like it’s a magic wand.
Pippa: Tyler, come on, you know teams already do that when it’s safe—sentiment, extraction, FAQ fallbacks. The trick is cheap confidence estimators.
Tyler: That ‘trick’ is a nontrivial research project. The article just says 'tune a classifier' and moves on.
Pippa: Okay, fair. But the prefix-caching section is legit—put your stable context at the front, let the cache do the work.
Tyler: Legit until your workload isn’t read-heavy. Cache hit rates crater if half your prompts are interactive agent loops where the user’s asking for the same thing phrased differently.
Pippa: That’s a workload caveat, though—not a strike against prefix caching itself.
Tyler: Fine. Cache layers section then—exact response cache, semantic cache, retrieval cache, tool result cache. Four kinds of cache, all needing tenant isolation, versioning, and similarity thresholds? That’s a distributed-systems tax most teams haven’t budgeted for.
Pippa: You’re not wrong about the tax. But the article’s point stands: don’t reprocess what you already know.
Tyler: Exactly. And that’s why the quality section is thin group therapy. They tell you to ask for shorter outputs and stop sequences without quantifying when brevity hurts accuracy. 'Do not pay for tokens the user won’t read'—great advice for a product owner, terrible validation criteria for an engineer.
Pippa: Tyler, you’re being unfair—they’re speaking to product people.
Tyler: Then it shouldn’t double as a production guide. The RAG section repeats the same mistake: 'control your context budget' without real guidance on how to triage relevance or deduplicate without losing signal.
Pippa: Alright, so what would you salvage?
Tyler: Three things the article nails. One: measure TTFT, P95/P99, queue time, cache hit rate—actually look at the levers you can pull. Two: tune batching against user latency SLOs, not raw throughput—continuous batching in vLLM is the best example they didn’t cite.
Pippa: Three: collapse superfluous LLM calls. A single structured prompt replacing three sequential steps is pure latency win. That part’s golden.
Tyler: Everything else is 'run offshore' advice. Routing to tiny models? Only after you’ve paid the evaluation tax. Cache layers? Only if your workload fits. Batch throughput? Only if it doesn’t blow TTFT.
Pippa: So the article’s strongest section is the measurement table and the batch-tuning section—rest is 'apply judiciously.'
Tyler: Judiciously, yeah, which is exactly what the article doesn’t say enough.
Pippa: You just don’t like any advice that isn’t seven pages of equations.
Tyler: I like advice that tells me the failure modes up front. This one buries them.
Pippa: Okay, fine. You want to put a number on it?
Tyler: Seventy-thirty the article’s blanket recommendations either backfire or don’t generalize—outside the three wins I named.
Pippa: And I’ll say forty-sixty those three wins move the needle for teams already running agents. The rest is noise.
Tyler: …Forty-sixty? Bold.
Pippa: I obviously didn’t sleep on this—literally. Wake up and measure your TTFT and P95 first.
Tyler: Fair enough. But given half the article’s tips need a PhD in ops, I’d say Build Next is in order.
Pippa: Go on, then—make it concrete.
Tyler: vLLM continuous batching plus Harmonia for RAG caching, and also the paper 'From Prefix Cache to Fusion RAG Cache'—both are open source, both ship, both have code.
Pippa: You just want me to run the stack so you can critique it.
Tyler: I want you to stop burning money while arguing about model routing.
Pippa: Fine. Wednesday, done.