Onyx: So Birgitta Böckeler just wrote this piece on local models for coding, and—I'm genuinely surprised by how unsentimental it is. She's not saying 'locals are ready, cloud is dead' or any of that. She's saying she ran models on her MacBook Pro, tested for a month, and here's what actually worked and what didn't.
Echo: Right. And the detail I appreciate is she's not cherry-picking wins. She's literally mapping the variable space—RAM, quantization, context window size, whether reasoning is on or off, tool calling quality. That's the honest move.
Onyx: Yeah. I mean, she starts by saying she'd been avoiding local models for years because the disappointment was always too high. But a month ago she dove back in because the claims were too loud to ignore. So this is her actual four-week experiment.
Echo: Mm-hm.
Onyx: She ran on two machines—M3 Max with 48GB RAM, M5 Pro with 64GB—and tested models between 15 and 25GB in size. The headline: they're runneable and fast, way faster than a year ago. But tool calling—which you need for agentic coding—still fails constantly, and quality is really hit-or-miss.
Echo: The speed part is real. I've seen the same thing in the last few months with MLX and newer quantization. But here's where I'd push back a little: she tested on Apple Silicon with unified memory, which is a pretty generous setup. Most of the industry is still on discrete GPUs.
Onyx: Fair. Though she does call that out—she notes on Apple Silicon, basically all RAM is accessible to the GPU, which is different from a lot of other setups. She's not hiding that.
Echo: Right, right. And the thing that actually got me is the variable map. She lists like thirteen different factors that all interact: RAM, model size, quantization level, architecture—MoE versus dense—context window, tool schema complexity, whether reasoning is on or off, the format you're running in, processing cores, memory bandwidth...
Onyx: Yeah, that's the thing. It's not 'use model X and you're done.' It's 'every variable talks to every other variable, and changing one breaks your assumptions about the others.' She even notes that on her automated eval setup, one model performed BETTER on the stronger machine—not just faster, but better code—and she can't fully explain why.
Echo: Exactly. That's the tell that the space is still messy. If the landscape were settled, you wouldn't see that kind of artifact.
Onyx: Okay, quick caveat here—like, we do a lot of deep-dive episodes where we re-derive claims from papers and benchmarks. This one, we're trusting Birgitta's read and the article itself, so a detail or two could be off. But the overall pattern is solid.
Echo: Fair. So what was actually the standout finding for you?
Onyx: The reasoning thing. She ran her automated setup with reasoning ON by default—all the models had chain-of-thought built in. But when she turned it OFF, the smaller models didn't just get faster, they actually performed the same or better. They were stuck in loops like 'Wait, actually, but wait, hmm...' and it was hurting them.
Onyx: Right? Like, reasoning is not a free win. It's task-dependent. And on a 30B model, it can be a liability.
Echo: That's honest. Most of the hype around reasoning models treats it as strictly better. The reality is it's a tradeoff—you get more compute, longer latency, and sometimes the model uses that extra compute to talk itself into a corner.
Onyx: And then tool calling. She says the models often fail at it—they emit malformed calls, wrong parameter names—but the harness can usually self-recover. So it's not a dealbreaker, but it's not seamless either.
Echo: That's the agentic bottleneck right there. If your model can't reliably emit the tool-call schema the harness expects, you're stuck doing the old copy-paste dance. And for teams that want actual agents, that's a non-starter.
Onyx: Yeah. Though she does say the self-recovery thing is interesting—the models catch themselves and fix it. That's not nothing.
Echo: True. And then there's the MoE win with Qwen. The Qwen three-point-six 35B MoE model gave her the best balance between capability and runnability. A 35B dense model would need way more RAM; the MoE version only activates a subset of weights at inference, so it stays under the ceiling.
Onyx: Right. And she notes that the 80B MoE—Qwen three Coder Next—actually solved tasks way better than the smaller models, but then crashed after a few more turns in the conversation. So the bigger model exists, but the RAM ceiling is real.
Echo: That's the product constraint. The model's good, the hardware's the bottleneck. So you're not picking the best model; you're picking the best model that fits your RAM.
Onyx: Exactly. And context window size is another sneaky one. She had to manually set it to 32K or 64K for agentic work—the runtime defaults were way too small. That eats RAM on top of the model weights through the KV cache, which grows as context gets longer.
Echo: Right. So you're not just fitting the model; you're fitting the model plus the conversation history plus the tool descriptions. All of that is in VRAM.
Onyx: Yep. The whole piece is basically saying: locals are viable now in a way they weren't a year ago, but it's not plug-and-play. You have to understand the variable stack, tune for your hardware, and accept that quality is going to be lower than Claude or GPT-5.
Echo: And the follow-up memo—which she mentions at the start—is going to be her actual experiences. So this is the framework piece. The next one is the 'here's what happened when I tried to build something' piece.
Onyx: Yeah. So this is useful. Not a 'go local' manifesto, just a sober map of the territory.
Echo: Exactly. And for teams that care about data privacy or want to avoid API costs, there's a real path now. It's just not as simple as downloading Llama and going.
Onyx: That's the whole thing. Okay, I'm calling it: I'd say seventy percent that this becomes a meaningful baseline for the 'can I run a local agent?' question by end of year. Birgitta's got credibility, and this is the kind of honest evaluation that usually sticks.
Echo: I'd take that bet. Though I'd push the timeline back a few months—the tool-calling problem is still real enough that most teams are going to hit it and bounce.
Onyx: Fair. Alright, that's episode six-oh-six. Thanks for walking through this one, Echo.
Echo: Yeah. Good piece. Honest piece.