Catalog and Semantic Layer Architecture

The Catalog Service, Semantic Layer, and Governance Service form the metadata management triad of the Data Plane. Together they provide schema discovery, business-friendly metric abstraction, data lineage, classification, and access policy enforcement.

The Catalog Service manages the platform's metadata catalog -- databases, tables, columns, schemas, lineage, and data classifications.

Core responsibilities:

• Data source registration and automated schema discovery
• Table and column metadata management with type information
• Data lineage tracking (upstream/downstream dependencies)
• Schema change detection and versioning
• Integration with OpenMetadata for metadata interchange
• Full-text search and discovery via tags and classifications

Schema discovery flow:

1. Data source registered (PostgreSQL, S3, Iceberg, etc.)
2. Catalog service connects via appropriate connector
3. Introspects schema: databases, tables, columns, types
4. Stores metadata in catalog database
5. Publishes CATALOG_UPDATED event to Kafka
6. AI service receives event, updates RAG context

Key APIs:

The Semantic Layer provides a business-friendly abstraction over raw database schemas. It translates business concepts (metrics, dimensions, relationships) into SQL.

Semantic model structure:

Semantic Model
  +-- Entities (e.g., "Customer", "Order", "Product")
  |    +-- Dimensions (e.g., "region", "category", "date")
  |    +-- Metrics (e.g., "total_revenue", "avg_order_value")
  |    +-- Relationships (e.g., Customer --has_many--> Order)
  |
  +-- Calculations
       +-- Simple: SUM(amount), COUNT(DISTINCT customer_id)
       +-- Derived: revenue_per_customer = revenue / customer_count
       +-- Time Intelligence: YoY growth, MTD, rolling 7-day average

Query translation example:

Semantic query:
  "total_revenue by region for Q4 2025"

Generated SQL:
  SELECT
    d.region,
    SUM(o.amount) AS total_revenue
  FROM orders o
  JOIN dim_geography d ON o.geo_id = d.id
  WHERE o.order_date BETWEEN '2025-10-01' AND '2025-12-31'
  GROUP BY d.region
  ORDER BY total_revenue DESC

The Semantic Layer caches computed metrics in Redis with tenant-scoped keys. Cache TTL depends on the metric's refresh frequency configuration.

The Governance Service enforces data governance policies:

Integration with Apache Polaris provides fine-grained access control at the Iceberg catalog level.

Related Sections

• Data Catalog -- Full catalog documentation
• Query Architecture -- How queries use catalog metadata
• AI Architecture -- How AI uses semantic models