By Andrew Tan
Why I'm writing this
Because AI is all the rage at the moment, some CTOs ask themselves: "Should AI replace half our data engineering team?"
That's the state of AI in data engineering right now. Everyone's publishing breathless content. Nobody's being specific. So here's my take on the topic:
What AI genuinely helps with
SQL generation is the clearest win. Copilot-style tools cut the time to write a first-draft analytical query by 50-70% for engineers with solid SQL fundamentals. You still need to review it. You still need to know what the answer should look like. But the blank-page problem is gone.
Schema documentation is dramatically faster. Getting from "we have 400 tables" to "we have documented 400 tables" used to take months of analyst time. With good LLM tooling, teams can get through this in weeks. The documentation isn't perfect, but it's good enough to be useful, which it often wasn't before.
Ad-hoc analysis has changed meaningfully for non-engineers. Business analysts who used to file tickets for "can you write me a query that…" can now get working answers to simple questions themselves. This is real productivity. It's also a meaningful reduction in interrupt-driven work for data engineering teams.
Code review drafts. Not a replacement for review, but catching the obvious stuff — unindexed joins, missing null checks, type mismatches — before a human looks at it saves time in aggregate.
These are real and they matter. I don't want to dismiss them.
What AI can't reliably handle
Here's where the gap between vendor claims and production reality opens up.
Schema evolution at scale
The hardest part of maintaining production pipelines isn't writing the code — it's knowing what to do when an upstream system changes a field type, deprecates a column, or starts sending data in a different format. This requires understanding the business logic behind the data, the downstream consumers, the historical context of why the field exists. An LLM that wasn't in the room when those decisions got made can't reliably reason about the right response. It'll give you something that looks right. It often isn't.
Stateful stream processing
A team can spend three months trying to get an LLM to correctly implement a windowed aggregation with late-arrival handling for their real-time fraud detection pipeline. The LLM could write the code. The code also runs. It produces wrong answers in edge cases that only show up in production, under specific ordering conditions, on days with unusual event volumes. Those bugs are the hard kind — they don't throw errors, they just silently corrupt your metrics. The model has no way to test its own output against the actual edge cases it will face.
Production failure recovery
When a Kafka consumer falls behind by 48 hours and you need to decide whether to replay, drop, or deduplicate — that's not a code generation problem. That's a judgment call that requires knowing your business, your SLAs, and the cost of each option. I've yet to see an LLM make that call correctly without significant human scaffolding.
A lead engineer at a cyber security company told me: "We got to about 70% automation on our standard ETL patterns. The last 30% is the stuff that actually breaks in production." He wasn't complaining. He understood why. But the 30% is what keeps data engineers employed.
The "80% automation" problem
Gartner published a prediction last year that 80% of data engineering work would be affected by 2027. I understand why they wrote it.
Here's the thing about 80%: the 80% they're talking about is scaffolding. Boilerplate. First drafts. The part that's genuinely 80% automatable for example is the part that was already relatively fast.
What remains is the 20% that takes 80% of the time — debugging why the data looks wrong, negotiating schema changes with upstream teams, reasoning about pipeline reliability under conditions nobody anticipated. That 20% is also the 20% where a wrong answer is expensive.
I'm not saying this to be pessimistic. The 80% matters. Freeing engineering teams from scaffolding is genuinely valuable. But the teams that plan for a world where this automation means fewer engineers are making a specific bet that the expensive problems will also get easier. They might. I'm not seeing evidence of it yet.
What I tell teams considering headcount reductions
Don't do it yet. Not because the technology isn't real, but because you're betting on the wrong variable.
The teams getting the most from AI tooling aren't the ones cutting headcount — they're the ones taking the same headcount and pointing it at harder problems. The engineers who used to spend their days on routine ETL work are now working on data quality frameworks, schema governance, real-time pipeline reliability. The output per engineer is higher. The quality of the output is higher. The team is harder to replace, not easier.
That's the story. AI is a productivity multiplier for data engineers. It's not THE data engineer.
A simple overview
I know I said I'd avoid the comparison table format. But this one is genuinely the clearest way to show it:
| Task | AI helps | AI struggles |
|---|---|---|
| SQL generation | First drafts, 50-70% faster | Complex logic with subtle business rules |
| Schema docs | First pass, weeks not months | Accurate semantics without business context |
| Ad-hoc analysis | Simple questions for non-engineers | Questions requiring cross-system context |
| Pipeline code | Boilerplate, standard patterns | Stateful logic, edge-case handling |
| Schema evolution | — | Almost entirely human judgment |
| Failure recovery | — | Requires business + operational knowledge |
| Production debugging | — | LLMs don't know your specific history |
The left column is real. The right column is why data engineering teams still exist.
Where layline.io fits
I'll be direct: the AI productivity gains I described above are easier to capture when your pipelines have explicit structure that LLMs can understand and extend.
At layline.io, we build pipelines with declarative configuration — the logic is in structured operators, not embedded in custom code (except for the casual Javascript or Python here and there and only where really necessary). That turns out to pair well with AI-assisted development. When an engineer asks an LLM to add a processing step, the LLM can reason about it clearly. When something breaks, the failure is in a known place rather than buried in bespoke code.
That's not why we built it that way. We built it that way because declarative pipelines are easier for humans to debug and maintain. The AI affinity turned out to be a side effect.
But it does mean that teams building on a structured foundation get more out of AI tooling than teams working in custom code. Something worth considering when you're making architectural choices that will matter in two years.
The question worth asking your team
Try this: pick your last five data incidents. For each one, ask whether an AI could have prevented it or diagnosed it faster.
For most teams the answer is "maybe 1 out of 5." The other four are problems an LLM can't reliably reason about — wrong business logic that is technically correct code, a schema change from an upstream team that nobody announced, an edge case in stream processing that only manifests at specific event volumes.
If you're evaluating AI tooling, that's your baseline. Not "will AI change data engineering" — of course it will. But "will AI eliminate the problems that actually hurt us?" That answer is no, not yet, and probably not without something changing that hasn't changed.
Andrew Tan is a serial entrepreneur and founder of layline.io, building enterprise data processing infrastructure that handles both batch and real-time workloads at scale.
