By Andrew Tan
Change Data Capture is the invisible layer enabling real-time analytics and event-driven systems — but most teams only think about it after their first production incident.
The Invisible Layer That Everything Depends On
Real-time dashboards. Event-driven microservices. Data lakes that stay current. Behind every one of these modern data architectures sits a component that most teams don't think much about: Change Data Capture.
CDC's job is simple enough — watch database transaction logs and emit events whenever data changes. New order? Event. Status update? Event. Customer deletion? Event. The concept is elegant, and when it works, it just works.
But there's a problem. CDC is the plumbing of modern data infrastructure: invisible when it functions, catastrophic when it fails, and somehow always an afterthought in architecture reviews. Teams spend weeks debating Kafka topologies and Spark configurations, then slap in a CDC connector with default settings and move on.
Six months later, the call comes. The dashboard is six hours behind. The inventory sync is showing yesterday's data. The CEO is asking why customers can buy products that don't exist. And nobody can figure out why — because the CDC connector is "healthy" according to the monitoring dashboard.
This pattern plays out across the industry with remarkable consistency. The issue isn't that CDC is fundamentally unreliable. It's that the gap between what teams assume it does and what it actually does is wide enough to hide production incidents until they become business problems.
What CDC Actually Does (And What Teams Assume It Does)
At its core, Change Data Capture watches your database transaction log and emits events whenever data changes. Insert a row? Event. Update a field? Event. Delete a record? Event. The concept is beautifully simple.
But the simplicity is deceptive. Here's what CDC actually captures versus what teams assume it captures:
| What teams assume | What actually happens |
|---|---|
| "Every change is captured immediately" | There's latency. Sometimes milliseconds, sometimes seconds, sometimes longer if the connector is backlogged. |
| "The events are in the same order as the transactions" | Not necessarily. Parallel replication, commit ordering, and eventual consistency can scramble sequences. |
| "Schema changes are handled gracefully" | Adding a column? Fine. Renaming one? Dropping one? Changing a type? Your CDC pipeline may need manual intervention. |
| "It's just a log tail, what could go wrong?" | Connector crashes, replication slot exhaustion, disk space issues on the source DB, network partitions... |
The gap between assumption and reality is where incidents breed.
The Three Failure Modes Nobody Talks About
After watching a dozen CDC implementations go sideways, I've noticed three failure patterns that don't get enough attention in the tutorials and vendor demos.
1. The Schema Drift Trap
Your application team adds a new column to the orders table. It's a harmless change — a nullable delivery_notes field. They deploy on Tuesday. By Thursday, your data warehouse has incomplete records because the CDC connector is still using the old schema and silently dropping the new field.
The worst part? The connector doesn't fail. It just produces events that are technically valid but practically wrong. Your data quality monitors don't catch it because the schema validator thinks everything is fine. You only discover the gap when someone asks why the delivery notes report is blank for half the week.
2. The Replication Slot Bomb
PostgreSQL users, this one's for you. CDC connectors use "replication slots" to track which WAL (Write-Ahead Log) entries they've processed. If your connector goes down — or even just slows down significantly — those slots hold onto log entries. The database can't reclaim that disk space.
I've seen teams wake up to production databases at 95% disk capacity because a flaky CDC connector was holding replication slots hostage. The fix is a manual cleanup job that feels terrifying to run at 2 AM. The prevention? Monitoring and alerting that most teams don't set up until after the first incident.
3. The Consumer Coupling Problem
CDC emits a firehose of events. Every microservice, analytics job, and data warehouse sync that cares about database changes taps into that stream. It's elegant and decoupled — until it isn't.
What happens when one slow consumer can't keep up? Backpressure propagates. The CDC connector buffers, then drops, then crashes. Or worse: it keeps running but falls behind, and your "real-time" pipeline has a 20-minute lag that nobody notices because the metrics dashboard shows "connector healthy."
The fix is usually some form of buffering (Kafka, Kinesis, a message queue) between the CDC source and the consumers. But now you've added latency and another piece of infrastructure to manage. The simple plumbing has become a complex subsystem.
Sizing for Reality, Not for Hope
Here's a fictional conversation:
Me: "How many transactions per second does your CDC need to handle?"
Them: "Oh, maybe a few hundred during peak."
Me: "And what's your biggest table?"
Them: "About fifty million rows."
Me: "What happens when you run a bulk update on that table?"
Them: "...We do those sometimes."
CDC connectors aren't sized for your average transaction volume. They're sized for your worst-case transaction volume. That quarterly data cleanup job that touches ten million rows? That generates ten million CDC events in a burst. If your connector can't handle the spike, you get lag, backpressure, or dropped events.
The teams that do this well plan for bursts from day one. They set up monitoring on replication lag, not just connector health. They test their failure modes: what happens if the connector restarts mid-bulk-update? What happens if the destination is down for an hour?
Design Decisions That Make CDC Manageable
CDC doesn't have to be a ticking time bomb. Here are the patterns I've seen work in production:
Separate CDC Infrastructure from Analytics Infrastructure
Don't run your CDC connector on the same cluster as your Spark jobs or your BI queries. When the analytics team runs a heavy join that saturates the network, your CDC events shouldn't suffer. Give CDC its own lane.
Idempotent Consumers Are Non-Negotiable
CDC events can be duplicated. Connectors restart, network partitions happen, at-least-once delivery is the default. If your downstream consumer can't handle "process this order update twice," you're going to have data corruption. Build idempotency in from the start.
Schema Registries Save Sanity
Use a schema registry (Confluent Schema Registry, AWS Glue, or similar) to track changes to your event schemas. When the application team changes a table, the schema change flows through the registry and your consumers can adapt programmatically instead of breaking silently.
Monitor What Matters
"Connector is running" is the wrong metric. Monitor:
- Replication lag (how far behind is the CDC from the database?)
- Event processing rate (are we keeping up with production?)
- Schema change events (did something change in the source we need to know about?)
- Dead letter queue depth (what couldn't be processed and why?)
Where layline.io Fits: CDC Without the Footguns
At layline.io, we've watched teams struggle with CDC enough that we built a dedicated Debezium Source Asset directly into the platform. The goal isn't to reinvent CDC — Debezium is excellent — but to wrap it in the reliability and observability that production systems need.
Instead of running a standalone connector that you have to babysit, layline.io gives you:
Visual pipeline design that includes CDC sources as first-class citizens. You see the data flow from database to destination on a single canvas. When something breaks, you know exactly where.
Built-in backpressure handling through Apache Pekko's actor-model streaming. When downstream systems slow down, layline.io throttles gracefully instead of dropping events or crashing connectors.
Unified retry and error handling across the entire pipeline. CDC events that fail to process don't vanish into a log file — they go through the same retry mechanisms as every other data source.
Schema-aware transformation that can adapt to changes in the source database without manual intervention. Add a column, rename a field, change a type — the pipeline adjusts instead of breaking.
The broader point: CDC is too important to be an afterthought. It deserves the same engineering rigor as the rest of your data infrastructure. Whether you use layline.io or build your own stack, treat CDC like the critical component it is — not like plumbing you can ignore until the basement floods.
Andrew Tan is a serial entrepreneur and founder of layline.io, building enterprise data processing infrastructure that handles both batch and real-time workloads at scale.
