CDC & Replication

Change Data Capture (CDC) enables real-time data replication from source databases to the Matih data platform via Airbyte's logical replication support.

Supported CDC Sources

Database	Replication Method	Requirements
PostgreSQL	Logical replication (pgoutput/wal2json)	`wal_level = logical`, publication + replication slot
MySQL	Binlog replication (GTID)	`binlog_format = ROW`, replication user
SQL Server	CT (Change Tracking)	Enable CT on database + tables
MongoDB	Change Streams	Replica set or sharded cluster
Oracle	LogMiner	Supplemental logging enabled

Configuring CDC

Step 1: Create Source with CDC Support

POST /api/v1/sources
{
  "name": "prod-postgres",
  "connectorType": "postgres",
  "connectionConfig": {
    "host": "db.example.com",
    "port": 5432,
    "database": "mydb",
    "username": "replication_user",
    "password": "..."
  }
}

Step 2: Enable CDC on the Connection

POST /api/v1/connections/{connectionId}/cdc
{
  "replicationSlot": "matih_slot",
  "publicationName": "matih_publication",
  "snapshotMode": "INITIAL",
  "heartbeatEnabled": true,
  "heartbeatInterval": 300000
}

Step 3: Verify CDC Status

GET /api/v1/connections/{connectionId}/cdc

Snapshot Modes

Mode	Behavior
`INITIAL`	Full snapshot on first sync, then CDC for subsequent changes
`INITIAL_ONLY`	Full snapshot only, no ongoing CDC
`WHEN_NEEDED`	Snapshot only when no valid offset exists
`NEVER`	Skip snapshot, start from current position (may miss historical data)

Sync Lifecycle with CDC

The connection FSM supports CDC-specific transitions:

CREATED → TESTING → TESTED → SCHEMA_DISCOVERED → CONFIGURED → SCHEDULED
                                                                    ↓
                                              SYNCING → COMPLETED / FAILED
                                                ↑          ↓
                                              RETRYING ←──┘

When CDC is configured, the sync mode is automatically set to CDC and Airbyte uses logical replication instead of full table scans.

RBAC

Permission	Roles
`connections:write`	DATA_ENGINEER, PLATFORM_ADMIN
`connections:read`	DATA_ENGINEER, DATA_ANALYST, DATA_SCIENTIST

DATA_ANALYST and DATA_SCIENTIST can view CDC configuration but cannot modify it.

Parquet Import Stream Ingestion