Jessica: Okay, so video MLLMs are still getting crushed by token count. You throw a five-minute video at an image tokenizer, sparse-sample it, and you've already lost the motion. VideoChat3 says: stop doing that.
Cathy: Right. The standard move is frame-by-frame encoding — each frame is basically an independent image. You get no temporal compression, no redundancy reduction, and the LLM has to stitch the motion story together after the tokens are already huge.
Jessica: Which is backwards.
Cathy: Exactly. VideoChat3 inverts it: model the spatiotemporal structure early, in the encoder, before you ever hand tokens to the LLM. That's the I3D-ViT move — Inflated 3D Vision Transformer. You're processing video patches as 3D spatiotemporal blocks, not 2D image patches.
Jessica: So the vision encoder is doing temporal modeling, not the language model.
Cathy: Yeah. And because you're modeling spatiotemporal structure, you can compress redundancy — adjacent frames have massive overlap, right? If the encoder understands that, it can throw away the redundant stuff early. You end up with way fewer tokens before the LLM even sees them.
Jessica: How much fewer?
Cathy: The paper doesn't give a raw token count comparison, which is annoying. But the throughput numbers suggest significant compression. The real win is streaming: they add adaptive frame resolution, so if motion is low, you drop spatial resolution. High motion, you keep it. That's the kind of thing that makes streaming actually feasible.
Jessica: Streaming is real-world. Autonomous driving, surveillance, live sports commentary — you can't wait for the full video. You need to respond as frames come in. And if you're doing adaptive resolution, you're not just being efficient, you're being smart about where efficiency matters.
Cathy: Right. The architecture is sound. But here's where I want to push: they're claiming 4B parameters and they say it outperforms larger open models like Qwen3-VL and Molmo2. That's a bold claim. Better architecture and better data can beat parameter count, but I'd want to see those benchmarks independently reproduced.
Jessica: Fair. But the data story is where I think they actually have a real move. They're not just throwing everything into one pile. VideoChat3-Academic2M for general video, VideoChat3-LV116K for long-form, VideoChat3-OL617K for streaming. Three datasets, three distinct scenarios.
Cathy: And they use multi-stage curriculum learning — general first, then long-form, then streaming. That's deliberate. You don't want task interference.
Jessica: Exactly. And the curriculum is the thing that actually matters for generalization. You can have a great architecture, but if your training signal is a mess, the model learns a mess.
Cathy: Yeah. The curriculum is load-bearing.
Jessica: But here's the thing that actually got me: they're releasing everything. Model weights, training code, training strategy, complete datasets, the data construction pipeline. That is not the move most labs make.
Cathy: No. Most papers are either closed or partially open — you get weights maybe, but no training code, no data, no recipe. You can't reproduce it. You can't improve it.
Jessica: Right. And VideoChat3 says: reproducibility is not a nice-to-have, it's the product. If you can't reproduce it, you can't build on it.
Cathy: Okay, so that's the thesis. The question is: does it actually ship? Does anyone use this? Or is it research-grade forever?
Jessica: That's the real test. A 4B video model that's efficient and open is genuinely useful for teams that don't have billions in compute budget. If this actually works at inference time, it could be the foundation for a bunch of stuff.
Cathy: The efficiency claim is testable. You can run it, measure latency, measure tokens per second, see if it actually holds up under production load. That's the bet I'd make: the architecture is sound, the data strategy is smart, but whether it's efficient at scale is still an open question.
Jessica: Yeah. And the streaming part especially — because streaming is not just about efficiency, it's about latency. If you're doing autonomous driving, you need sub-hundred-millisecond latency. Does VideoChat3 actually hit that?
Cathy: Right. The benchmarks they report are standard video understanding tasks. Real-time latency under production constraints is a different test.
Jessica: Okay, so the move is: go grab the code and the datasets from Hugging Face, spin up a model, and actually test it. The full pipeline is open, so you can do that.
Cathy: Yeah. That's the whole point. You don't have to wait for a paper. You don't have to reverse-engineer it. You can just… build with it.
Jessica: The model weights are on Hugging Face under MCG-NJU/VideoChat3, the code is on GitHub at MCG-NJU/VideoChat3, and the datasets are all there too. If you're building a video understanding system and you don't have a billion-dollar budget, this is your starting point.
Cathy: And the data construction pipeline is open, which means you can actually understand how they built the datasets. That's not trivial. Most papers, the data is a black box. Here, you can see the process.
Jessica: Which means the community can improve it. You can add your own video types, your own scenarios, your own curriculum. The foundation is reproducible, so iteration is possible.
Cathy: That's the rare part. Most video MLLMs are not built for iteration. They're built to be published and then left alone.
Jessica: VideoChat3 is built to be built on. That's the actual product.
Cathy: Yeah. Okay. I'm in.