Data engineering · 10 min read
When change data capture becomes a modernisation problem
A production checklist for CDC covering snapshots, ordering, deletes, schema change, recovery, reconciliation, monitoring, and ownership.
CDC is easy to demonstrate and hard to operate
Change data capture becomes a modernisation problem when the organisation needs more than a stream of inserts and updates. A production service must establish a correct starting state, preserve the order that matters, carry deletes, survive restarts, accommodate schema change, expose lag, reconcile source and target, protect the source, and assign operational ownership.
CDC can shorten the distance between an operational change and its downstream use. It does not automatically provide synchronous replication, a business-event contract, global ordering, or exactly-once processing from source to consumer.
If the team cannot explain how the pipeline restarts, reconciles, and handles a breaking schema change, it has a feed—not yet a production data product.
When CDC is worth using
| Good fit | Benefit | Poor fit or warning |
|---|---|---|
| Low-downtime database or platform migration | Copy the baseline while changes continue, then drain and reconcile before cutover | The business requires a synchronous write path or has no acceptable consistency window |
| Fresher analytics and operational reporting | Reduce repeated full extracts and make committed changes available sooner | A daily batch already meets the requirement and the operating cost of CDC is not justified |
| Search indexes, caches, and read models | Keep downstream projections updated when eventual consistency is acceptable | Records lack a stable identity for updates and deletes |
| Bounded downstream automation | Trigger replayable processing from committed changes | The automation needs business intent that raw table mutations do not express |
| Staged coexistence during modernisation | Supply the target while consumers move in waves | Both systems accept writes without a tested ownership and conflict model |
Use “near-real-time” only with a measured freshness objective. Latency depends on source load, transaction size, network, connector capacity, target apply capacity, and outages. A useful target is specific, such as a p95 source-commit-to-consumer delay, rather than a label with no threshold.
Join the initial load and change stream at one position
The initial copy and the incremental stream form one recovery protocol. A safe conceptual sequence is:
- Record or establish the native source-log position.
- Retain or capture changes from that point.
- Take a consistent snapshot or full load.
- Apply changes captured while the load was running.
- Continue incremental capture from the exact handoff position.
- Reconcile the baseline and subsequent deltas.
- Declare steady state only after lag and mismatches meet the release criteria.
Do not bulk-copy first and switch CDC on afterward without a protected common position. Changes committed in that gap can be missed permanently. Product handoff semantics differ, so test snapshot collisions, long transactions, restart behaviour, and the transition into streaming under representative load.
Five design questions teams often miss
1. What order must be preserved?
Ordering is usually guaranteed within a source key or stream partition, not across the whole database. Use a stable business or source key, preserve the source position and transaction metadata, and ensure target parallelism cannot reorder changes for the same record. Do not infer global commit order from consumer arrival time.
2. What does a delete mean downstream?
Decide how hard deletes, soft deletes, key changes, truncates, and retention are represented and applied. A target that ignores delete events becomes quietly wrong even while insert and update metrics look healthy.
3. How will schemas change safely?
Version the change-event contract, prefer compatible additive changes, and test type changes, renames, key changes, precision changes, and dropped fields against consumers. Do not assume every source DDL statement will propagate correctly across every engine pair.
4. How will the system recover?
Persist offsets outside an ephemeral worker and retain source logs for the longest credible outage plus repair time. Assume failures can replay events, then make target writes idempotent or prove the delivery boundary end to end. Rehearse worker loss, target outage, source failover, expired log positions, backlog drain, and re-snapshot.
5. How will correctness be demonstrated?
Message delivery is not proof of correct target state. Reconcile counts by table or business date, insert/update/delete volumes, stable chunk checksums, important aggregates, freshness, checkpoint continuity, and sampled records. Every material mismatch needs an owner and a repair path.
Monitor more than “job running”
A green connector can be stale, incomplete, or unable to recover. Production monitoring should cover:
- end-to-end freshness, plus separate capture and target-apply lag;
- changes entering and leaving the pipeline, backlog, queue pressure, and disk spill;
- last-event or heartbeat age for low-volume sources;
- connector state, restart count, offset failures, and dead-letter volume;
- source transaction-log retention, oldest required position, and free storage;
- target apply and schema-compatibility failures;
- reconciliation mismatches and the age of unresolved exceptions.
Set thresholds from the business freshness objective and tested peak profile, not from vendor dashboard defaults.
Protect the source and the data path
Log-based capture still has production cost. Snapshots read tables, validation issues queries, connectors retain and decode logs, and outages create backlog. Measure source CPU, read I/O, log generation and retention, storage, network, locks, and application latency. Cap concurrency and define stop conditions before running heavy snapshots or validation.
Use a dedicated least-privilege capture identity, encrypted connections, managed secrets, restricted access to offsets and schema history, and data minimisation at the capture boundary. Apply retention and masking rules to event logs, errors, backups, and dead-letter records—not only the destination table.
A practical do and do-not checklist
| Do | Do not |
|---|---|
| Define a measurable freshness and recovery objective | Call the pipeline real time without a tested latency bound |
| Join full load and CDC at a recorded source position | Leave an unprotected gap between bulk copy and capture |
| Model updates, deletes, key changes, and replay | Treat CDC as an append-only insert feed |
| Version schemas and coordinate DDL | Let source changes break consumers without compatibility checks |
| Monitor lag, backlog, log retention, errors, and mismatches | Monitor only whether the process is alive |
| Reconcile continuously and rehearse recovery | Assume delivery means the target is correct |
How CDC creates measurable benefit
Baseline the current process and report the change after a representative operating period. Useful measures include source-to-consumer freshness percentiles, actual cutover downtime, source workload removed from periodic extracts, tested recovery time, mismatch rate and resolution time, pipeline availability, and operator interventions.
Do not publish generic improvement percentages without project evidence. A defensible result states the measured target and its conditions: for example, that eligible changes reached a governed reporting layer within an agreed p95 freshness threshold while source impact, reconciliation exceptions, and recovery remained inside approved limits.
Primary references
- Debezium: PostgreSQL connector, snapshots, offsets, events, and monitoring
- Apache Kafka: ordering, compaction, and tombstones
- PostgreSQL: publications and replica identity
- AWS DMS: CDC best practices and latency limitations
- AWS DMS: source-to-target validation
- Confluent: schema evolution and compatibility