Query Engine Architecture

SQL Input (from AI service, BI service, or direct API)
  |
  v
1. PARSE
   - Parse SQL into AST (Abstract Syntax Tree)
   - Validate syntax correctness
   |
   v
2. VALIDATE
   - Resolve table and column references against catalog
   - Check tenant has access to referenced tables
   - Validate column types for expressions
   |
   v
3. OPTIMIZE
   - Analyze query plan for optimization opportunities
   - Push down predicates where possible
   - Estimate query cost (rows, bytes, time)
   |
   v
4. AUTHORIZE
   - Check row-level security policies (governance-service)
   - Apply column masking rules for sensitive data
   - Verify user has query:execute permission
   |
   v
5. EXECUTE
   - Submit to Trino via JDBC connection
   - Set search_path / catalog to tenant's data sources
   - Stream results back with timeout enforcement
   |
   v
6. CACHE
   - Hash query + parameters for cache key
   - Store results in Redis with tenant-prefixed key
   - TTL based on query type (60s for dashboards, 0 for ad-hoc)
   |
   v
7. RETURN
   - Paginate results (default 1000 rows, max 10000)
   - Include query metadata (execution time, rows scanned)
   - Publish QUERY_COMPLETED event to Kafka

Query Engine --> Trino Coordinator --> Trino Workers
                                          |
                    +---------------------+---------------------+
                    |                     |                     |
              PostgreSQL             S3/ADLS              Iceberg
              Connector              Connector             Connector
                    |                     |                     |
              Tenant metadata       Lakehouse data        Lakehouse data
              (schemas, config)     (Parquet, ORC)        (versioned tables)

// Set tenant catalog context before query execution
Connection conn = trinoDataSource.getConnection();
conn.setCatalog(tenantId + "_catalog");
conn.setSchema(targetSchema);

Key format: {tenant_id}:query-engine:result:{query_hash}

Cache policy:
  - Dashboard queries: 60s TTL (refreshed on access)
  - Scheduled reports: 300s TTL
  - Ad-hoc queries: No caching (TTL = 0)
  - AI-generated queries: 30s TTL

public QueryCostEstimate estimateCost(String sql, String tenantId) {
    // Uses Trino's EXPLAIN (TYPE DISTRIBUTED) to get:
    return QueryCostEstimate.builder()
        .estimatedRows(trinoEstimate.getOutputRowCount())
        .estimatedBytes(trinoEstimate.getOutputSizeInBytes())
        .estimatedTimeMs(trinoEstimate.getCpuCost())
        .dataSourcesAccessed(trinoEstimate.getConnectors())
        .build();
}