Database Connectors

Database connectors extract data from relational and NoSQL databases. They support schema discovery, multiple sync modes, and type-safe extraction with automatic Iceberg schema mapping.

PostgreSQL

The PostgreSQL connector supports Full Refresh, Incremental, and CDC (via logical replication / WAL) sync modes. It is one of the most widely used connectors on the platform.

Configuration

{
  "name": "production-postgres",
  "connectorType": "postgres",
  "connectionConfig": {
    "host": "pg.example.com",
    "port": 5432,
    "database": "analytics",
    "username": "matih_readonly",
    "password": "********",
    "ssl_mode": "require",
    "schemas": ["public", "sales"],
    "replication_method": {
      "method": "CDC",
      "replication_slot": "matih_slot",
      "publication": "matih_pub"
    }
  }
}

Configuration Fields

CDC Prerequisites

To use CDC mode with PostgreSQL:

1. Set wal_level = logical in postgresql.conf
2. Create a publication: CREATE PUBLICATION matih_pub FOR ALL TABLES;
3. Create a replication slot: SELECT pg_create_logical_replication_slot('matih_slot', 'pgoutput');
4. Grant the connector user replication privileges: ALTER USER matih_readonly REPLICATION;

Type Mapping

MySQL

The MySQL connector supports Full Refresh, Incremental, and CDC (via binlog) sync modes.

Configuration

{
  "name": "mysql-orders",
  "connectorType": "mysql",
  "connectionConfig": {
    "host": "mysql.example.com",
    "port": 3306,
    "database": "orders_db",
    "username": "matih_reader",
    "password": "********",
    "ssl_mode": "required",
    "replication_method": "CDC"
  }
}

Configuration Fields

CDC Prerequisites

To use CDC mode with MySQL:

1. Enable binary logging: log_bin = ON in my.cnf
2. Set binlog format: binlog_format = ROW
3. Set binlog row image: binlog_row_image = FULL
4. Grant replication privileges: GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'matih_reader';

MongoDB

The MongoDB connector supports Full Refresh and CDC (via change streams) sync modes.

Configuration

{
  "name": "mongo-users",
  "connectorType": "mongodb",
  "connectionConfig": {
    "connection_string": "mongodb+srv://matih_reader:********@cluster0.example.net/",
    "database": "user_data",
    "auth_source": "admin"
  }
}

Configuration Fields

CDC Prerequisites

MongoDB CDC requires a replica set or sharded cluster. Standalone MongoDB instances do not support change streams. The connector user needs the readAnyDatabase and read roles.

SQL Server

The SQL Server connector supports Full Refresh, Incremental, and CDC sync modes.

Configuration

{
  "name": "sqlserver-erp",
  "connectorType": "mssql",
  "connectionConfig": {
    "host": "sqlserver.example.com",
    "port": 1433,
    "database": "ERP",
    "username": "matih_reader",
    "password": "********",
    "ssl_method": "encrypted_trust_server_certificate",
    "schemas": ["dbo"],
    "replication_method": "CDC"
  }
}

Configuration Fields

CDC Prerequisites

To use CDC mode with SQL Server:

1. Enable CDC on the database: EXEC sys.sp_cdc_enable_db;
2. Enable CDC on each table: EXEC sys.sp_cdc_enable_table @source_schema = 'dbo', @source_name = 'orders', @role_name = NULL;
3. Ensure SQL Server Agent is running (required for CDC cleanup jobs)

Oracle

The Oracle connector supports Full Refresh, Incremental, and CDC (via LogMiner) sync modes.

Configuration

{
  "name": "oracle-finance",
  "connectorType": "oracle",
  "connectionConfig": {
    "host": "oracle.example.com",
    "port": 1521,
    "sid": "ORCL",
    "username": "matih_reader",
    "password": "********",
    "encryption": {
      "encryption_method": "client_nne",
      "encryption_algorithm": "AES256"
    },
    "schemas": ["FINANCE"]
  }
}

Configuration Fields

Other Database Connectors

The platform supports additional database connectors through Airbyte.

For detailed configuration of any connector, consult the Airbyte connector documentation (opens in a new tab) for the specific connector version deployed in your tenant.