Justy: Okay, so Databricks just dropped Lakeflow at their summit, and they're calling it 'agentic data engineering' — which, I mean, that's a lot of buzzwords in one place.
Cody: Yeah, 'agentic' is definitely the word of the year. But honestly, the pitch is interesting — AI that writes and maintains your data pipelines so you don't have to.
Justy: Right? Like, imagine your data engineer never sleeps and never complains about YAML. Anyway — you just got back from San Francisco, right? How was the summit?
Cody: Exhausting. I think I walked twenty miles between sessions. But Lakeflow was everywhere — they're clearly betting big on this. The idea is that instead of manually configuring pipelines, you describe what you want in natural language, and the platform handles the rest.
Justy: So it's like an AI data engineer assistant? That's… actually useful. I've seen teams spend weeks just keeping pipelines from breaking.
Cody: Exactly. They've got this thing called Genie Code and Lakeflow Designer — you basically type 'ingest this API, join it with that table, and output to a dashboard' and it generates the pipeline. Then it monitors and adjusts itself.
Justy: Wait — it adjusts itself? Like, if a source schema changes, it just… fixes it?
Cody: That's the claim. They call it 'Genie ZeroOps' — automated scaling, error recovery, drift detection. No more paging someone at 3 AM because a field went from string to integer.
Justy: Okay, that's huge. I mean, I'm skeptical about 'agentic' meaning 'it works perfectly,' but if it handles even half of the operational overhead, that's a win. What about connectors?
Cody: Lakeflow Connect — over a hundred built-in connectors. SaaS APIs, databases, file systems. And they have this thing called Zerobus Ingest that lets data producers push events without setting up Kafka. It's basically a managed ingestion service.
Justy: Huh. So they're trying to remove every excuse for not having clean, real-time data. What's the catch? There's always a catch.
Cody: The catch is the same as always — it's Databricks. You're in their ecosystem. If you're already on the lakehouse, great. If not, you're migrating. And the 'agentic' part probably works best on standard patterns — weird legacy stuff might still need a human.
Justy: Fair. But for teams that are all-in on Databricks, this could genuinely reduce headcount on pipeline maintenance. That's a real product story.
Cody: Yeah. And they've got Spark Declarative Pipelines now — batch and streaming, SQL and Python, with real-time support. So you're not locked into one paradigm.
Justy: So if someone wants to try this — what's the move?
Cody: It's available now. There's a free tier, documentation's live at databricks dot com slash lakeflow. If you're at the summit, there are hands-on labs.
Justy: Alright, I'm curious enough to spin it up. Cody, you're coming with me on this one?
Cody: As long as I don't have to write any YAML.
Justy: Deal. Alright — that's Lakeflow. Go play with it. We'll compare notes next week.