By Andrew Tan
The expenses nobody puts in the migration budget — and why the real price of going real-time has nothing to do with software licenses
The budget that didn't survive first contact
A VP of Engineering I know budgeted $180,000 for his team's batch-to-streaming migration. That was twelve months ago. Last time we spoke, the project had consumed $640,000 and they were still six weeks from production.
What happened? Not fraud. Not scope creep in the traditional sense. They simply failed to account for the costs that don't appear in vendor quotes: the six-week delay while they hired a Kafka engineer who understood exactly-once semantics. The three months spent running batch and streaming in parallel because nobody trusted the new pipeline yet. The emergency consulting engagement when their streaming aggregation produced different numbers than the batch report and the CFO noticed.
The software itself was cheap. The hidden costs ate them alive.
I've watched this pattern repeat across companies of every size. Teams budget for infrastructure and licenses. They don't budget for uncertainty, rework, and the operational tax of maintaining two systems while one replaces the other. By the time they realize what's happening, the project is either over budget or under-delivered — sometimes both.
Here's what actually costs money when you move from batch to streaming.
Cost #1: The talent you don't have yet
Batch engineering and stream engineering are related the way carpentry and furniture making are related. Same raw material, completely different craft.
Your existing team knows cron schedules, table scans, and the comforting finality of a job that starts, runs, and finishes. Streaming asks them to think in event time, manage unbounded state, and debug systems that never stop running. Some of your engineers will adapt quickly. Others won't — not because they're bad engineers, but because distributed stream processing is genuinely difficult and not everyone wants to specialize in it.
This creates a hidden cost in three forms:
Hiring: A senior stream processing engineer in London or New York costs between $160,000 and $220,000 base salary right now, plus the four-month average time-to-hire for that specialty. If you need two of them, that's nearly half a million in salary before they've written a line of production code.
Training: Your existing engineers need to learn new concepts: watermarking, consumer lag, partition skew, stateful operators, at-least-once versus exactly-once. These aren't afternoon workshop topics. They're months of hands-on learning where productivity is lower than normal and mistakes are more expensive than usual.
Attrition: Some of your best batch engineers will leave during the migration — not because they can't learn streaming, but because they didn't sign up to become distributed systems specialists. They liked the data work. They'll go somewhere that still does it the way they enjoy.
The budget line item for "talent" in most migration plans covers training. It rarely covers hiring delays, lost productivity, or unexpected turnover.
Cost #2: The parallel operation period
Nobody talks about this enough. You cannot simply turn off batch and turn on streaming. Not if you value your job.
For some period — typically three to six months, occasionally longer — you'll run both systems. The batch pipeline keeps producing the reports everyone trusts. The streaming pipeline runs alongside it, producing results that theoretically should match but often don't, at least not at first.
This means double the infrastructure. Double the monitoring. Double the alerts. And a team of engineers spending their days reconciling two sets of numbers instead of building new features.
One e-commerce company I worked with ran parallel systems for eight months. Their batch stack cost roughly $4,200 per month in cloud compute. Their streaming stack cost $7,800 per month. For eight months, they paid both. That's $96,000 in infrastructure alone — never mind the engineering time spent investigating why the streaming count of Tuesday's orders was 347 off from the batch count.
The parallel period isn't optional. It's insurance. But like all insurance, it's expensive, and most teams underestimate the premium.
Cost #3: The data archaeology
Your batch pipelines contain years of accumulated business logic. Somewhere in a 400-line Python script that runs at 2 AM is a join condition that exists because of a pricing exception from 2019. Nobody documented why it's there. The person who wrote it left in 2021. But if you remove it, the revenue numbers shift by 0.3% and finance sends angry emails.
Migrating to streaming means understanding every one of these artifacts. You can't simply port the code. The logic needs to be reimplemented for continuous event processing, which means you first have to understand what it does and why. This is data archaeology — tedious, slow, and impossible to estimate accurately because you don't know what you'll find until you start digging.
A financial services firm I advised spent five weeks on a single pipeline. The streaming implementation took three days. Figuring out why the batch version produced a specific corner-case output took the other thirty-two days. The business logic was encoded in a stored procedure written by three different people over four years, with comments like "fix for Q2 bug" and no further explanation.
Cost #4: The operational complexity tax
Batch pipelines fail visibly. A job crashes. You get an alert. You fix it. You rerun it. Everyone understands what happened.
Streaming pipelines fail subtly. Consumer lag builds over hours. State stores grow until they hit memory limits. Watermarks drift and suddenly your windowed aggregations are dropping late events. By the time you notice, you've been producing slightly wrong results for half a day.
The operational tooling is different too. You're not just monitoring whether a job finished. You're monitoring latency distributions, throughput slopes, backpressure signals, and state store sizes. Your existing runbooks don't apply. Your existing alerts don't catch the new failure modes.
Building this operational maturity takes time and mistakes. The first time your streaming pipeline silently drops 2% of events for six hours, you'll invest heavily in better observability. That's a necessary cost. But it's almost never in the initial budget.
Cost #5: The opportunity cost nobody measures
While your best engineers are debugging partition rebalancing and reconciling batch versus streaming outputs, they aren't doing other work. Feature requests pile up. Technical debt accumulates. Competitors ship things your team would have built if they weren't neck-deep in migration.
This is the hardest cost to quantify and the easiest to ignore. There's no invoice for it. But it's real.
One SaaS company paused all new data product development for nine months during their streaming migration. When they finished, they'd built a technically impressive real-time pipeline — but their primary competitor had shipped three analytics features in the same period and gained market share. The migration was a technical success and a strategic delay.
Why we keep underestimating
Part of the problem is vendor messaging. Streaming platforms sell the destination: real-time insights, instant reactions, competitive advantage. They don't advertise the journey: the hiring, the parallel systems, the archaeology, the operational learning curve.
Another part is optimism bias. Every engineering team believes they'll be the exception. Their code is cleaner. Their team is smarter. Their requirements are simpler. Sometimes that's true. Usually it isn't.
The result is a persistent gap between the budgeted cost and the actual cost. I've seen ratios of 2:1, 3:1, even 5:1. Not because anyone was dishonest — because the real costs are invisible until you've already committed.
How to budget honestly
You can't eliminate these costs, but you can account for them. Here's how I advise teams to think about it:
Add a 40% buffer to infrastructure estimates. The parallel period, the testing environments, the shadow deployments — they all add compute and storage you won't predict precisely.
Budget for six months of dual operation minimum. If you finish earlier, celebrate. If you don't, you won't be explaining overruns to your CFO.
Hire or contract one streaming specialist before you start, not after you get stuck. The cost of bringing them in early is high. The cost of bringing them in after three months of false starts is higher.
Accept that some pipelines should stay batch. Not everything benefits from real-time. Daily reporting, historical analytics, ML training pipelines — these are often batch-appropriate workloads that don't justify the migration cost. Be explicit about what you're not migrating.
A different way to think about the transition
The teams that handle this well share one trait: they don't view it as a migration. They view it as adding a capability.
Instead of "we're moving from batch to streaming," they say "we're adding streaming where it creates value, and keeping batch where it still works." This sounds like semantics, but it changes the economics completely. You're no longer committed to moving everything. You can evaluate each pipeline on its own merits: latency requirements, complexity, business value, migration cost.
Some pipelines move. Some don't. The ones that move justify their own investment. The ones that stay don't generate unnecessary cost.
This is where a unified platform matters. If you're running separate tools for batch and streaming, every pipeline faces pressure to migrate because maintaining two platforms is expensive. If you can run both models on the same platform — same workflows, same team, same operational approach — the pressure disappears. You add real-time streaming where it earns its keep and leave batch alone where it's already working.
That's the approach we built into layline.io. Not because batch is bad — it's often exactly right — but because forcing teams to choose one approach and abandon the other creates artificial cost and risk. The teams that sleep well at night are the ones that didn't try to boil the ocean.
The bottom line
The hidden cost of switching from batch to streaming isn't the software. It's the everything else: the people you need to hire, the systems you need to run in parallel, the legacy logic you need to excavate, the operational maturity you need to build, and the opportunity you lose while you're focused on infrastructure instead of product.
Budget for it. Account for it. Be honest about which pipelines actually need to move and which don't.
The goal isn't to be real-time everywhere. The goal is to be real-time where it matters, without bankrupting yourself to get there.
What's next
If you're planning a batch-to-streaming migration, start with an honest audit. List your top ten pipelines. For each one, ask: what's the actual cost of latency? What's the estimated migration effort? What's the operational complexity add?
If the numbers don't justify the move for a given pipeline, leave it alone. Focus your energy on the two or three where real-time creates measurable business value.
For teams evaluating platforms, the Community Edition of layline.io is free to explore. You can prototype a streaming pipeline alongside your existing batch workflow and see what the operational reality looks like before you commit the budget.
Andrew Tan is a serial entrepreneur and founder of layline.io, building enterprise data processing infrastructure that handles both batch and real-time workloads at scale.
