By Andrew Tan
The expensive truth about modern data stacks
Spend enough time around data platform teams and you hear the same story. A company builds out its "modern data stack" — warehouse, processing layer, orchestrator — and everything looks clean on the architecture diagram. Then the warehouse bill starts to climb. Ingestion jobs fail more often than anyone expected. And every time something breaks, it takes half a day to figure out whether the problem is in the load, the reshape, the orchestrator, or the warehouse itself.
At some point, someone on the team says the quiet part out loud: "I think we built a really expensive integration tool by accident."
They are usually right.
The category error
A data warehouse is a query and storage engine. It is optimized for one thing: answering analytical questions fast over large datasets.
A data pipeline is a movement and processing runtime. It is optimized for something different: getting data from where it is to where it needs to be, in the right shape, at the right time, reliably.
Those are different jobs. But in the last decade, we've quietly asked the warehouse to do both.
It started innocently. Warehouses got better at loading data. Then they got stored procedures. Then dbt turned SQL into a processing layer. Then orchestrators started triggering warehouse queries to move data between tables. And before anyone named it, the warehouse had become the default integration layer.
The result is predictable. The warehouse is excellent at analytics. It is mediocre at integration. And when you force it to do integration at scale, you pay for it in three currencies: cost, reliability, and architectural fragility.
What goes wrong when the warehouse becomes the pipeline
The compute bill becomes a surprise
Warehouse compute is priced for analytical queries. Analysts run a few big queries, wait for results, and go make decisions. The compute is bursty and human-paced.
Integration workloads don't look like that. They run continuously or on tight schedules. They move millions of rows. They run the same conversions over and over. They don't pause to let humans read dashboards.
When you run this kind of workload inside a warehouse, the meter spins differently. It is common for a "simple" hourly sync to consume more credits than the entire analytics workload. Not because the warehouse is bad, but because it's the wrong engine for the job.
Failures become opaque
A pipeline has a clear job: take data from A, transform it, deliver it to B. When it fails, you want to know which step failed and why.
When the warehouse is the pipeline, failure is distributed across layers. Was the load slow because the warehouse was overloaded? Did the orchestrator lose its connection? Did the reshape query hit a timeout? Is the data wrong because of the source, the conversion, or a change to the warehouse execution plan?
Debugging becomes archaeology. You dig through query history, orchestrator logs, and warehouse metrics, trying to reconstruct what actually happened. The tools are all there. The clarity isn't.
Latency is whatever the warehouse decides
If your pipeline is a series of warehouse queries, your latency is bounded by warehouse scheduling. A query waits in a queue. It compiles. It runs. Maybe it gets preempted. Maybe it scales up. Maybe it doesn't.
For batch analytics, this is fine. No one cares if a nightly report finishes at 3 AM or 3:15 AM.
For operational use cases, it's not fine. Fraud detection, inventory updates, customer-facing dashboards — these need minutes or seconds, not warehouse-queue time. When the warehouse is your pipeline, you inherit its pace. And its pace is designed for analysts, not operations.
Lock-in deepens
The more integration logic lives inside the warehouse, the harder it becomes to leave. Your rewrites are in warehouse-specific SQL dialects. Your orchestration is tied to warehouse sessions. Your data quality rules run as warehouse queries. Even your cost visibility is warehouse-shaped.
This isn't a conspiracy. It's just what happens when one tool becomes responsible for too many jobs. The migration cost grows until it feels easier to stay unhappy than to leave.
What clean separation looks like
The fix isn't to throw out the warehouse. The warehouse is good at what it does. The fix is to let it do what it does and stop asking it to do everything else.
In practice, that usually means two platforms, not one:
Integration and orchestration runtime
This is where data moves, gets reshaped, gets validated, and gets routed to the right consumers. It also schedules pipelines, retries failures, enforces dependencies, and triggers downstream work — both inside the platform and in external systems. It runs on an engine designed for continuous data flow, not query latency.
Warehouse
This is where data is stored and queried. It receives clean, ready-to-query data from the integration layer. It doesn't worry about how the data got there, when the next load arrives, or what to do if a job fails. It just answers questions.
Logically, you can still think of integration and orchestration as separate concerns. Operationally, they often belong in the same runtime. A pipeline that can move data but can't schedule itself, retry itself, or trigger the next step is only half useful. The best platforms combine both.
When these concerns are separated from the warehouse, each tool gets simpler. The integration layer is optimized for throughput and reliability. The orchestrator is optimized for dependency management and failure recovery. The warehouse is optimized for query performance.
Most importantly, problems stay in their lane. When ingestion fails, you look at the integration runtime. When a report is wrong, you look at the warehouse. When a job doesn't run, you look at the orchestrator — which, in a clean setup, is part of the same runtime that moves the data.
When warehouse-as-pipeline is actually fine
I don't want to overstate this. For some teams, the warehouse-as-pipeline pattern works fine.
If you're small, your data volumes are low, your reshaping is simple, and your latency requirements are "tomorrow is fine," then keeping everything in one place is a reasonable tradeoff. The operational simplicity is worth more than the architectural purity.
The problems start when the pattern keeps scaling past its natural limit. A team that outgrows it usually knows. The bills get weird. The failures get mysterious. The idea of adding a real-time use case becomes a multi-month project instead of a configuration change.
The question isn't whether the pattern is bad. The question is whether it's still the right pattern for where you are now.
The migration path nobody takes
Most teams imagine this separation as a rip-and-replace project. It doesn't have to be.
The better approach is to extract the movement layer first. Pick one data source. Instead of loading it directly into the warehouse and then reshaping it there, move it through a dedicated integration runtime first. Clean it. Validate it. Then write the clean data to the warehouse.
The warehouse doesn't change much. The analysts keep querying the same tables. But now those tables are fed by a pipeline that is designed for feeding tables.
Once one source is moved, the pattern repeats. Source by source. Pipeline by pipeline. Over time, the warehouse stops being the integration hub and becomes what it was meant to be: the analytics hub.
Teams that do this successfully don't start with the hardest pipeline. They start with a boring one. The boring pipelines teach you the pattern without the risk. The hard pipelines get easier once the pattern is in place.
Where layline.io fits
I'll be direct: this is the architectural bet behind layline.io.
We built a data processing platform that handles the integration and orchestration layer — both batch and streaming — without making the warehouse do the heavy lifting. Pipelines move data, reshape it, validate it, and deliver it. They also schedule themselves, retry on failure, enforce dependencies, and trigger downstream workflows inside layline or in external systems.
The warehouse stores the data and queries it. Each tool does its own job.
Because layline handles both batch and streaming in the same runtime, you don't end up with one tool for your hourly loads and another tool for your real-time events. Same workflows. Same observability. Same team. And because orchestration is built in, you don't need a separate orchestrator sitting on top, coordinating between layline and everything else.
That's not a pitch for everyone. If your warehouse-as-pipeline setup is working and your bills are sane, you don't need us. But if you're staring at a tripled warehouse bill and wondering how a "simple" sync got so expensive, the separation we're describing is probably what you're actually looking for.
The question to ask your team
Pick your three most expensive warehouse workloads. Not the biggest analytical queries — the ones that run all day, moving and reshaping data.
Ask: are these workloads answering business questions, or are they just getting data into a shape where it can answer business questions?
If the answer is the second one, you've got integration work running in an analytics engine. That's not a moral failing. It's a very common architecture. But it's also a very fixable one.
The warehouse is a powerful tool. It just isn't the only tool.
Andrew Tan is a serial entrepreneur and founder of layline.io, building enterprise data processing infrastructure that handles both batch and real-time workloads at scale.
