By Andrew Tan
The audit that never finds the real problem
Here's a scene that plays out in regulated industries every quarter:
The compliance team runs their governance scan. The data catalog shows classification tags on every warehouse table. The lakehouse has column-level security policies. The BI tool enforces row-level access. Everyone signs off. The auditors leave happy.
Meanwhile, a CDC feed is replicating customer PII from a transactional database to an analytics cluster. Nobody classified the stream itself — they classified the destination table, which is fine, except the stream passes through a staging topic where three different services subscribe. One of them is an AI enrichment job that sends fragments of the data to an external LLM API. The API call doesn't show up in the data catalog because the catalog only catalogs what lands, not what moves.
The governance program is impeccable at rest. It's nearly blind in motion.
This isn't a tooling failure. It's a category error. We've built governance around the assumption that data lives in tables and files, and that if we control the endpoints, we control the story. But modern pipelines don't work like that. Data is in constant motion — across brokers, through transformations, into model training sets, out to partner APIs — and most governance frameworks haven't caught up.
Why at-rest governance misses the hard cases
Batch-era governance made sense for batch-era architecture. Data moved in scheduled loads. You knew when it arrived. You could scan it, classify it, and apply policy before anyone queried it. The pipeline was basically a delivery truck, and governance happened at the loading dock.
Streaming and CDC broke that model. Data moves continuously. It gets joined in flight. It gets enriched by external services. It gets filtered, split, and routed to multiple consumers before it ever touches a governed destination. By the time it lands, the sensitive parts may have already leaked, been copied, or sent somewhere the catalog can't see.
The gap shows up in specific, painful ways:
Classification after landing is often too late. If a CDC stream carries unmasked credit card numbers through three intermediate topics before they reach the warehouse, the warehouse column-level policy is a Band-Aid on a bleed that happened upstream.
Batch governance tools don't understand streaming semantics. A data catalog can tell you what's in a table. It can't tell you that a streaming join is pulling PII from one topic and correlating it with behavioral data from another, creating a new composite dataset that nobody reviewed.
Lineage breaks at the seams. Most lineage tools trace table-to-table relationships. They don't trace the API call that enriches a record mid-pipeline, or the model that trains on a snapshot of the stream, or the reverse ETL job that pushes derived data back to a CRM. The map looks complete until you follow a record through its actual lifecycle.
Retention policies ignore movement-time copies. You set a 90-day retention on the warehouse. Great. But the stream consumer that materialized a view in Elasticsearch? The S3 bucket that the pipeline writes to for replay? The partner API that received a copy? Retention is only as strong as its weakest replica, and most governance programs don't know where the replicas are.
What governance-in-motion actually means
Moving governance into the pipeline doesn't mean replacing your data catalog or your lakehouse security model. It means extending policy enforcement to the places where data is actually moving.
Here are the four capabilities that matter:
Policy at the boundary
Instead of classifying data after it arrives, classify it when it enters the pipeline. A CDC feed from a customer database should carry its classification tags with it — PII, financial data, health records — and those tags should persist through every reshape, join, and routing decision. If a downstream consumer tries to send tagged data to an unapproved destination, the pipeline should block it, not log it and move on.
This sounds obvious, but most pipelines don't do it. The metadata that matters for governance — classification, consent flags, retention requirements — is usually stripped out during normalization or stored in a separate catalog that the runtime doesn't consult.
Approval gates for high-risk flows
Some data movements shouldn't happen without a second look. A pipeline that starts replicating a new table to an external analytics tool. A stream that begins including a newly added column with geolocation data. An AI workflow that wants to send customer transcripts to a third-party model.
These aren't failures. They're normal operations. But they're also the moments where governance risk concentrates. The right model isn't to block everything and wait for a ticket. It's to let low-risk flows proceed automatically and flag high-risk ones for approval — with the approval gate embedded in the pipeline itself, not in a separate workflow tool that the engineer may or may not remember to use.
Retention and deletion that follow the data
When a customer exercises their right to be forgotten, or when a retention period expires, the request needs to reach every copy of the data — not just the warehouse table. That includes stream replays, materialized views, model training snapshots, and partner API caches.
In practice, this means the pipeline runtime needs to track where data was sent and maintain the mapping between a record's identity and its replicas. Most streaming platforms don't do this. They treat each message as independent, with no memory of where it came from or where it went. Governance in motion requires a different model: messages carry identity, and the runtime maintains provenance.
Lineage that includes transformations, not just tables
Table-to-table lineage is useful for batch pipelines. For streaming and CDC, it's incomplete. You need to know that a particular enrichment step appended third-party data, that a join correlated two datasets that were supposed to stay separate, or that a filter silently dropped records that should have been retained for compliance.
This means lineage can't be a post-hoc scan of query logs. It has to be built into the runtime, capturing what each step did to the data as it happens.
The rollout path: start with interfaces
Nobody rebuilds their governance model in one project. The teams that succeed start small and expand.
The right place to begin is the pipeline boundary — the points where data enters or leaves your system. CDC feeds from production databases. API calls to external services. Reverse ETL jobs that push data back to operational tools. These are the highest-risk, highest-visibility interfaces, and they're where governance failures show up first.
Pick one interface. Add classification tagging at the source. Add a policy check before the data exits. Add lineage tracking for that one flow. Prove it works. Then expand to the next interface.
The teams I see doing this well share one trait: they treat governance as a runtime concern, not a documentation concern. The policy isn't written in a wiki and hoped for. It's enforced by the pipeline, tested in CI, and versioned with the code. When the policy changes, the pipeline changes. When the pipeline changes, the policy is revalidated.
Where this gets harder than it sounds
I want to be honest about the obstacles, because "just move governance into the pipeline" is easy to say and hard to do.
Existing tools aren't built for it. Most data catalogs, security scanners, and lineage tools were designed for batch warehouses. They expect scheduled scans, not continuous streams. They expect tables, not topics. Extending them to cover motion usually means custom integration work that the vendor doesn't support.
Performance matters. Adding classification checks, policy lookups, and lineage logging to a streaming pipeline adds latency. For high-throughput flows, the overhead has to be negligible — which means policy decisions need to be cached, evaluated asynchronously, or pushed to the edge of the pipeline where they don't block the critical path.
Organizational ownership is fuzzy. Data governance is often owned by a compliance team that doesn't write pipeline code. Platform engineering owns the runtime but doesn't set policy. Getting these teams to agree on what "governance in motion" means, who maintains it, and who gets paged when it breaks is often harder than the technical implementation.
The standards are still forming. There's no universal protocol for attaching governance metadata to streaming records. No standard API for pipeline policy enforcement. The vendors are building their own models, and they don't interoperate well. If you're betting on a specific approach, you're partly betting on which vendor's model wins.
Where layline.io fits
I won't pretend we solve every part of this. At layline.io, we've focused on the runtime layer: a processing engine for both batch and streaming that can carry metadata through transformations, enforce policies at pipeline boundaries, and maintain lineage across the full flow.
The bet is that governance shouldn't be a separate system sitting next to your pipeline, parsing logs and hoping it catches everything. It should be part of the pipeline itself — embedded in the same runtime that moves the data, enforced by the same code paths that transform it, visible in the same observability layer that tells you whether the pipeline is healthy.
This matters because the alternative is fragmentation. One tool for cataloging, another for streaming, another for batch, another for lineage, another for policy management. Each integration point is a seam where governance can leak. The teams I talk to are tired of seams.
We're not the only ones building in this direction. Confluent is pushing stream governance. Airbyte is framing sovereignty as an architectural problem. The whole market is moving toward the same conclusion: data in motion needs the same controls as data at rest. We're building our piece of that stack.
The question to ask your governance team
Next time your compliance team signs off on a clean audit, ask them this:
"Can you show me where customer PII goes between the production database and the warehouse? Not the endpoints — the path. Every topic, every enrichment call, every replica."
If the answer is a diagram with two boxes and an arrow, you don't have governance. You have hope.
The good news is that fixing it doesn't require ripping everything out. It requires moving the control point from the destination to the journey. Start at the boundaries. Add policy to the pipeline. Make lineage part of the runtime. The tools are getting better, the patterns are getting clearer, and the teams that figure this out first will have a genuine operational advantage — not just cleaner audits, but fewer incidents, faster recovery, and data infrastructure they can actually trust.
If you're thinking through governance for streaming or CDC pipelines, get in touch. We're working with teams on exactly this problem — and the solutions are more practical than they first appear.
Andrew Tan is a serial entrepreneur and founder of layline.io, building enterprise data processing infrastructure that handles both batch and real-time workloads at scale.
