CDC pipeline (deep dive)

A connector is not enough: the CDC chain in production is made of a source connector, a schema registry, transformations and sink connectors, with dedicated lag monitoring.

Problem

The basic CDC pattern describes reading the transaction log. In production several choices arise: initial snapshot blocking or not, binary schema (Avro) or JSON, partition by primary key or by table, handling of row deletion (Kafka tombstone), DDL migration handling.

Forces

• Initial snapshot of a 500 GB table can take days and block the connector.
• DB schemas evolve — a schema registry is needed to avoid breaking consumers.
• DELETEs are represented by tombstones (key + null value) that must be preserved.
• CDC lag can drift silently if not measured.
• A DDL migration (ALTER TABLE) can block the connector if not anticipated.

Solution

Build the chain in five stages: (1) source connector Debezium PostgreSQL / MySQL / Oracle; (2) schema registry Confluent / Apicurio for Avro / Protobuf; (3) Single Message Transforms (filter, mask PII, route by key); (4) sink connectors to target (lake, ElasticSearch, other DB) or custom consumers; (5) monitoring of lag with debezium_metrics_milli_seconds_behind_source. The initial snapshot must be configured (mode initial, schema_only, or no_data depending on usage).

Full pipeline

Source DB (Postgres)
  │ logical replication slot
  ▼
┌─────────────────────────────────────────────────────┐
│ Debezium PostgreSQL Connector                       │
│   - reads WAL via pgoutput plugin                   │
│   - emits raw change events                         │
└─────────┬───────────────────────────────────────────┘
          │ Avro key+value
          ▼
┌─────────────────────────────────────────────────────┐
│ Kafka Connect Worker                                │
│   - SMT: ExtractNewRecordState                      │
│   - SMT: MaskField for PII                          │
│   - SMT: OutboxEventRouter                          │
└─────────┬───────────────────────────────────────────┘
          │
          ▼
┌─────────────────────────────────────────────────────┐
│ Kafka topics: db.public.orders, db.public.invoices  │
│   key = primary key (partitioning)                  │
└─────┬────────────────────────────┬──────────────────┘
      ▼                            ▼
 Consumer (EDI hub)        Sink: ElasticSearch
                            (Debezium ES sink)

EDI implementation

Stellantis case (publicly cited at Kafka Summit conferences): SAP ECC on Oracle 19c, Debezium Oracle LogMiner on tables VBAP (order line), LIKP (delivery) → Kafka Streams transformations that assemble an OrderConfirmed from deltas → EDI consumer publishing an EDIFACT ORDRSP over AS4 to the supplier. End-to-end latency ~ 2 s vs the previous 6 h batches. Schema registry is crucial: adding a SAP column does not stop the chain.

Anti-patterns

• No schema registry — a breaking schema breaks all consumers.
• Initial snapshot without pacing — source DB saturates.
• Tombstones ignored by consumer — DELETEs don't propagate.
• Overly restrictive Single Message Transform filter — silent data loss.
• No lag monitoring — chain drifts without alert.

Related patterns

• Change Data Capture — synthetic version.
• Transactional Outbox (deep dive) — CDC on outbox table is the standard combo.
• Pipes and Filters — Kafka Connect SMT is an instance.
• Event Message — produced output.

Sources

• Debezium — Architecture documentation. debezium.io/documentation/reference/stable/architecture.html
• Confluent — Kafka Connect & Schema Registry. docs.confluent.io/platform/current/schema-registry/index.html
• Kreps J. — I Heart Logs: Event Data, Stream Processing, and Data Integration, O'Reilly 2014.