Connector Management

Connectors are the bridge between the MATIH platform and external data sources. The tenant service manages the lifecycle of data source connectors, including credential storage, connectivity testing, schema discovery, and usage monitoring. Each tenant configures their own set of connectors, which are then consumed by the query engine, data pipeline service, and catalog service.

Connector Architecture

Tenant Service                       Data Plane Services
+----------------------------+       +----------------------------+
| Connector Management       |       | Query Engine               |
|   - Create/Update/Delete   | ----> |   - Execute queries        |
|   - Test connectivity      |       |   - Connection pooling     |
|   - Store credentials      |       +----------------------------+
|   - Schema discovery       |       | Pipeline Service           |
+----------------------------+       |   - ETL jobs               |
         |                           |   - Incremental loads      |
         v                           +----------------------------+
  Kubernetes Secrets                 | Catalog Service            |
  (encrypted credentials)           |   - Schema indexing        |
                                     |   - Lineage tracking       |
                                     +----------------------------+

Supported Connector Types

Category	Type	Description
Relational	PostgreSQL	Open-source relational database
Relational	MySQL	Popular open-source RDBMS
Relational	SQL Server	Microsoft relational database
Relational	Oracle	Enterprise relational database
Cloud Warehouse	Snowflake	Cloud-native data warehouse
Cloud Warehouse	BigQuery	Google Cloud analytics warehouse
Cloud Warehouse	Redshift	AWS cloud data warehouse
Cloud Warehouse	Databricks SQL	Unified analytics platform
NoSQL	MongoDB	Document-oriented database
NoSQL	DynamoDB	AWS key-value and document store
File Storage	S3	AWS object storage
File Storage	Azure Blob	Azure object storage
File Storage	GCS	Google Cloud Storage
Streaming	Kafka	Distributed event streaming
API	REST	Generic REST API connector
API	GraphQL	GraphQL endpoint connector

Connector Entity

Field	Type	Description
`id`	UUID	Connector identifier
`tenantId`	UUID	Owning tenant
`name`	String	Display name
`type`	ConnectorType	Database type (e.g., `POSTGRESQL`, `SNOWFLAKE`)
`host`	String	Connection host or endpoint
`port`	Integer	Connection port
`database`	String	Database name
`schema`	String	Default schema
`credentialSecretName`	String	Kubernetes secret name for credentials
`sslEnabled`	Boolean	Whether SSL/TLS is required
`sslMode`	String	SSL mode (`verify-full`, `verify-ca`, `require`)
`status`	ConnectorStatus	`ACTIVE`, `INACTIVE`, `ERROR`, `TESTING`
`lastTestedAt`	Instant	Last connectivity test timestamp
`lastTestResult`	String	Last test result message
`metadata`	Map	Additional connector-specific properties
`createdAt`	Instant	Creation timestamp
`updatedAt`	Instant	Last update timestamp

Connector Lifecycle

Create Connector

POST /api/v1/tenants/{tenantId}/connectors
Authorization: Bearer {admin_token}
Content-Type: application/json

{
  "name": "Production Warehouse",
  "type": "SNOWFLAKE",
  "host": "acme.snowflakecomputing.com",
  "port": 443,
  "database": "ANALYTICS",
  "schema": "PUBLIC",
  "credentials": {
    "username": "matih_reader",
    "password": "encrypted_value",
    "warehouse": "COMPUTE_WH",
    "role": "MATIH_READER"
  },
  "sslEnabled": true,
  "metadata": {
    "account": "acme",
    "region": "us-east-1"
  }
}

Processing steps:

Validate connector configuration
Create Kubernetes secret for credentials in the tenant namespace
Test connectivity to the data source
If connectivity succeeds, save connector with ACTIVE status
Trigger schema discovery for catalog indexing

Response (201):

{
  "id": "770e8400-e29b-41d4-a716-446655440000",
  "name": "Production Warehouse",
  "type": "SNOWFLAKE",
  "host": "acme.snowflakecomputing.com",
  "status": "ACTIVE",
  "lastTestedAt": "2026-02-12T10:00:00Z",
  "lastTestResult": "Connection successful",
  "schemasDiscovered": 5,
  "tablesDiscovered": 142
}

Credential Storage

Credentials are never stored in the tenant service database. They are persisted as Kubernetes secrets:

apiVersion: v1
kind: Secret
metadata:
  name: connector-770e8400
  namespace: matih-acme
  labels:
    matih.ai/connector-id: "770e8400-..."
    matih.ai/connector-type: "snowflake"
type: Opaque
data:
  username: <base64>
  password: <base64>
  warehouse: <base64>
  role: <base64>

The SecretEncryptionService ensures that credentials are encrypted at rest using the Kubernetes encryption provider.

Connectivity Testing

Test Endpoint

POST /api/v1/tenants/{tenantId}/connectors/{connectorId}/test
Authorization: Bearer {admin_token}

Response:

{
  "connectorId": "770e8400-...",
  "success": true,
  "latencyMs": 145,
  "serverVersion": "Snowflake 7.34.1",
  "message": "Connection successful",
  "testedAt": "2026-02-12T10:05:00Z"
}

Failed test response:

{
  "connectorId": "770e8400-...",
  "success": false,
  "latencyMs": null,
  "serverVersion": null,
  "message": "Connection refused: verify host and port",
  "errorCode": "CONNECTION_REFUSED",
  "testedAt": "2026-02-12T10:05:00Z"
}

Common Error Codes

Error Code	Description
`CONNECTION_REFUSED`	Host unreachable or port closed
`AUTHENTICATION_FAILED`	Invalid credentials
`SSL_HANDSHAKE_FAILED`	TLS certificate validation error
`DATABASE_NOT_FOUND`	Specified database does not exist
`PERMISSION_DENIED`	Credentials lack required permissions
`TIMEOUT`	Connection timed out
`DNS_RESOLUTION_FAILED`	Hostname could not be resolved

Schema Discovery

When a connector is created or updated, the service triggers schema discovery to index the available databases, schemas, tables, and columns.

Discovery Flow

Tenant Service               Catalog Service
     |                              |
     |  POST /connectors/{id}/discover
     |  (async)                     |
     |----------------------------->|
     |                              |  Connect to data source
     |                              |  Query information_schema
     |                              |  Index tables and columns
     |                              |  Detect data types
     |                              |  Estimate row counts
     |  Discovery complete          |
     |<-----------------------------|

Discovery Results

GET /api/v1/tenants/{tenantId}/connectors/{connectorId}/schemas

{
  "connectorId": "770e8400-...",
  "schemas": [
    {
      "name": "PUBLIC",
      "tables": [
        {
          "name": "ORDERS",
          "type": "TABLE",
          "columns": 12,
          "estimatedRows": 1500000,
          "sizeBytes": 524288000
        },
        {
          "name": "CUSTOMERS",
          "type": "TABLE",
          "columns": 8,
          "estimatedRows": 50000,
          "sizeBytes": 10485760
        }
      ]
    }
  ],
  "discoveredAt": "2026-02-12T10:01:00Z"
}

Connector Management API

Method	Endpoint	Description
`GET`	`/api/v1/tenants/{id}/connectors`	List all connectors
`GET`	`/api/v1/tenants/{id}/connectors/{connectorId}`	Get connector details
`POST`	`/api/v1/tenants/{id}/connectors`	Create connector
`PUT`	`/api/v1/tenants/{id}/connectors/{connectorId}`	Update connector
`DELETE`	`/api/v1/tenants/{id}/connectors/{connectorId}`	Delete connector
`POST`	`/api/v1/tenants/{id}/connectors/{connectorId}/test`	Test connectivity
`POST`	`/api/v1/tenants/{id}/connectors/{connectorId}/discover`	Trigger schema discovery
`GET`	`/api/v1/tenants/{id}/connectors/{connectorId}/schemas`	Get discovered schemas
`PUT`	`/api/v1/tenants/{id}/connectors/{connectorId}/credentials`	Update credentials

Connector Limits by Tier

Tier	Max Connectors	Connector Types
Free	2	PostgreSQL, MySQL only
Professional	10	All standard types
Enterprise	Unlimited	All types including custom

When a tenant exceeds their connector limit, the creation endpoint returns 403 Forbidden with a message indicating the plan limit.

Monitoring and Health Checks

Connectors are periodically tested for health. A scheduled task runs every 15 minutes to verify connectivity:

Status	Condition
`ACTIVE`	Last test succeeded
`WARNING`	Last test succeeded but latency exceeded threshold
`ERROR`	Last test failed
`INACTIVE`	Connector disabled by user

When a connector enters ERROR status, the tenant admin receives a notification with diagnostic details.

Next Steps

API Reference -- all tenant management endpoints
Data Catalog -- how connectors feed the data catalog
Query Engine -- how connectors are used for query execution

Billing DNS Management