By Andrew Tan
Schema drift and upstream breaking changes are the number one cause of silent data failures. Here's how to stop getting blindsided.
The field that broke everything
A fintech company I know had a payment processor integration running beautifully for two years. Hundreds of thousands of transactions per day, solid pipeline, happy finance team. Then one morning the reconciliation reports started producing nonsense. Revenue figures were off by orders of magnitude. Negative counts where there should have been positive ones.
The root cause took six hours to find. Their payment processor had silently changed a field — amount — from a string to an integer. No version bump. No migration guide. No email. They just pushed it. The pipeline's conversion logic had been doing string operations on that field, slicing cents off, parsing currency codes, and when it suddenly received integers, it didn't error. It kept running and produced numbers that looked plausible enough to pass automated checks but were completely wrong.
Six hours of engineer time. Two days of finance team reconciliation. One very uncomfortable conversation with the CFO. All because a vendor changed a field type.
I've seen variations of this story more times than I can count. The source system changes without warning. The pipeline doesn't crash. It does something worse: it keeps running and produces garbage.
Why this doesn't show up in your monitoring
Here's the counterintuitive part. Most pipelines are designed to catch processing failures, not schema failures.
You get alerts when a job doesn't run. You get alerts when latency spikes. You get alerts when the destination database rejects a write. What you don't usually get alerts for is when the input changed shape and your processing logic silently went sideways.
The reason is structural. Schema validation tends to be an afterthought. Teams write their pipeline logic first, worry about validation later, then never actually implement it because the pipeline is already running and touching it feels risky.
So you end up with a system that's battle-hardened against infrastructure failures and completely blind to data contract violations.
Three classes of upstream change
Not all upstream changes are equally dangerous. Three categories:
Additive changes are the friendly ones. A new optional field appears. An array gains an extra item type. The source system grows without breaking existing structure. Your pipeline generally handles these fine — you just ignore whatever you don't need. Low risk.
Breaking changes are the obvious enemy. A required field disappears. A data type changes. A field gets renamed. These tend to cause hard failures, which is actually fine. Your pipeline crashes, you get an alert, you fix it. Painful but detectable.
Silent changes are the worst. These are changes that don't cause failures. The field is still there. The type is still technically compatible. But the semantics shifted. A status field that used to contain "active" and "inactive" now contains "enabled" and "disabled". Your null checks still pass. Your row counts still look normal. Your dashboards still populate. Everything is wrong.
Silent changes are the ones that end up in CFO conversations.
Where most pipelines fail
The instinct is to add more validation at the processing layer. Check types before you convert. Assert that required fields are present. Run shape checks at ingestion.
This helps, but it misses the core problem. Validation at the processing layer catches type errors. It doesn't catch semantic drift. It doesn't catch the case where a field is present, correctly typed, and completely wrong in meaning.
The real defense is at a different layer entirely: the contract between the source system and your pipeline.
A contract defines not just the structure of the data but the expectations around it. Field X is a string representing a currency-denominated amount in minor units. Field Y is an enum with exactly these four values. Field Z is a timestamp in UTC, never null, always within the last 90 days.
When you define that contract explicitly and test against it on every ingestion, you catch silent changes before they corrupt downstream data. You catch the case where amount is still a string but now represents the full amount rather than the amount in minor units. You catch the case where the enum gains a fifth value your pipeline has never seen.
Contract testing at the pipeline layer
Contract testing is well-established in microservices (Pact is the most common framework), but it's underused in data pipelines. The basic idea is simple: define the shape and semantics of your input as a formal contract, then run tests against that contract every time data flows through.
In practice, this means three things.
A schema spec that goes beyond field types. Include expected value ranges, cardinality constraints, the relationship between fields. Not just amount: integer but amount: positive integer, max 10,000,000, represents cents.
A monitoring layer that runs these checks continuously, not just at pipeline startup. One failed check per run is fine. Ten thousand failed checks on 500,000 records is your silent-change alarm.
An alerting threshold calibrated to your data. If 0.1% of records historically have null transaction_id (anomalous but real), set your alert at 1%. When the null rate hits 1%, something changed. When it hits 15%, something definitely changed.
Is it worth implementing? Yes, with one caveat: write your contracts at the point of integration, not after the fact. Retrofitting contracts onto an existing pipeline is painful because you have to reverse-engineer what you assumed was true. The time to document your expectations is when you first build the integration, while the knowledge is fresh.
Defending against each class
Additive changes: just ignore them. Build your pipeline to extract only what you need. Don't break on unexpected fields.
Breaking changes: fail loud and fast. A hard crash with a clear error message is better than silent corruption. Strict validation at ingestion on the fields you depend on.
Silent changes: this is where contract testing earns its keep. Track distributions, not just structure. If the distribution of status field values suddenly shifts from 90% "active" / 10% "inactive" to 50/50, something changed upstream. Maybe it's a legitimate business shift. Maybe it's a semantic change in how they classify accounts. Either way, you want to know.
How layline.io handles this
The challenge with most pipeline frameworks is that schema changes require pipeline changes. You update a field mapping, redeploy the job, hope nothing else breaks. The feedback loop between "upstream changed" and "pipeline adapted" is hours or days.
In layline.io, the processing model separates the logical work (what you want to do with the data) from the physical format (what shape the data arrives in). Schema evolution, new fields, renamed fields, type changes, can be handled through configuration rather than code. Your logic operates on logical fields mapped from the physical schema, so when the physical schema changes, you update the mapping without rewriting the pipeline.
This doesn't eliminate the need for contract testing. You still want to know when something unexpected arrives. But it dramatically reduces the blast radius. An upstream change that would have required a pipeline rewrite and redeployment becomes a configuration update that takes minutes.
It also means you can run both batch and streaming integrations against the same source system with the same schema handling. When the payment processor changes a field, you fix it once — not separately for your real-time fraud detection pipeline and your nightly reconciliation batch job.
The practical takeaway
Start with the contracts you don't have. Pick your three most critical upstream integrations, the ones where a silent data change would cause the most damage, and write down what you actually expect from each field. Not just the type. The range, the allowed values, the semantic meaning.
Then build monitoring that catches deviations from those expectations. Not perfect, not broad, just the highest-stakes integrations first.
Most teams operate without explicit contracts until something breaks. The teams that operate with them find out about upstream changes on their own terms, not at 3 AM with corrupted data.
Andrew Tan is a serial entrepreneur and founder of layline.io, building enterprise data processing infrastructure that handles both batch and real-time workloads at scale.
