Vince: Okay, okay—so there’s this repo that’s basically a one-way ticket to a PhD position if you follow the README closely enough. Train LLM From Scratch, written in plain PyTorch, no transformers, no peft, no trl—just raw attention and hand-written loops.
Ava: Right. And it magically fits a billion parameters on a single GPU—sure.
Vince: That’s not the half of it—this is end-to-end. Raw text to tokenized pile to base model to SFT to reward model to PPO to GRPO to chat interface. One repo, one idea.
Ava: One idea is a loop: tokenize, next-token prediction, tweak loss, repeat until it does what you want. Still needs hardware.
Vince: Fair—but the student appeal is huge. If you’re trying to land a PhD spot, showing you trained your own model from scratch is … kind of perfect?
Ava: Stop—this is a teaching repo, not a product. Still excited to see it.
Vince: I’m more than excited—I’m giddy. Look at the GPU table: 13M model fits on a T4, a billion needs an A100. That’s not a bug, that’s a feature. It’s honest about the ladder.
Ava: On a free Colab T4? You’ll max out around two billion. A billion-parameter run is still fantasy land.
Vince: Yeah, yeah, but the point is you can START on a T4. For most of us, that’s the only step that matters. Tomorrow you need a bigger card, you rent time or save up—still better than chasing GPUs as a student.
Ava: Okay, okay. And the 5090 finally pushes into the billion-parameter zone—if you’re willing to drop two grand and pull three-fifty watts.
Vince: Exactly. Hardware gate, not model gate. That’s the clarity I love.
Ava: Sure. Now, the post-training half—the SFT, reward model, DPO, PPO, GRPO—all hand-coded inside one tiny transformer? That’s the part that actually interests me.
Vince: I see. So it’s not just pretraining—it’s the full RLHF pipeline baked into one repo. Researchers get a compact stack to hack, not just a toy.
Ava: Compact, yes. Maintainable? Only if the dual config systems don’t bite you.
Vince: Right—two configs: plain Python constants for legacy pretrain, JSON plus CLI overrides for everything else. It’s editable, import-friendly, and it ships smoke tests that finish on a CPU in seconds.
Ava: Smoke tests are the lifeboat. Still—two config systems means two places to mis-read the README.
Vince: True, but the smoke configs exist for exactly that reason. Run a five-second check, then scale up.
Ava: Fair. Now the data side—four streams, r50k_base, HDF5 files, loss masks that train only on assistant tokens. Okay, I’ll admit it: the peek at the first ten token IDs is cute.
Vince: Ten integers that decode straight back to text—yeah, that’s the sausage. The whole thing is tokenization plus loss masking plus ‘keep tweaking the loss.’
Ava: If you squint. Still—the loss mask is the bit that keeps the model from parroting prompts.
Vince: Agreed. And the color-coded legend in the README means every block has a job, not just a name—nice touch for students.
Ava: …Still a teaching tool. But if we’re handing out gold stars, I’ll give one for the chat template.
Vince: Oh, come on—this is the rare repo that actually lets you go full loop. Pretrain base, SFT it, build a reward model, run PPO or GRPO, then talk to it. All on your own GPU.
Ava: On your own big-ish GPU. Or one you rent.
Vince: Exactly. Now—where does this sit in the race? It’s not a frontier model, it’s a training lab. Open-weight snapshot as of June 2026: DeepSeek V4 Flash leads the pack, Z.ai’s GLM-5.2 is pushing agentic coding, and the frontier tier is still Claude/GPT/ Gemini.
Ava: So this is four-to-twelve weeks behind the top closed systems, not miles behind. Still useful pressure.
Vince: And useful for students and open labs who want to hack post-training without buying a data-center cluster.
Ava: …I’ll grant you that. Still—remember when you swore off GraphQL?
Vince: Oh, that’s right, you dragged that up. Yes, I still stand by it. But this? This is different.
Ava: Okay, okay. Build Next: how do we actually run this?
Vince: Clone the repo, editable install—pick the extras you need. git clone the link in the show notes cd train-llm-from-scratch pip install -e "." if you’re doing training, or "." if you want the full tour.
Ava: And for a 13M pretrain run on a T4:
Vince: python scripts/prepare_pretrain_data.py --split train --num_shards 1 --out data/pile_train.h5 python scripts/train_transformer.py
Ava: Then chat it with the Streamlit panel.
Vince: There you go. Now go update your PhD application and tell them you trained an LLM from scratch on one GPU.
Ava: …Good luck with that. I’m filing this under “teaching aid that will outlive most PhD timelines.”