Part IV: Data Plane Services

Chapter 12: AI Service and Conversational Analytics

The AI Service is the largest and most complex microservice in the MATIH platform, implementing the conversational analytics engine that transforms natural language questions into data insights. Built on Python/FastAPI with 984 source files, it orchestrates 80+ agent types, manages multi-turn conversations, generates SQL from natural language, and delivers results through real-time streaming.

Learning Objectives

Understand the modular architecture with 7 feature-flagged modules
Master the AgentOrchestrator and MultiAgentOrchestrator patterns
Learn the Text-to-SQL pipeline from NL parsing through SQL validation
Explore the BI Analytics orchestrator state machine
Configure the multi-provider LLM Router with tenant-level routing strategies
Integrate the Context Graph for agent thinking traces and semantic search

Details

Estimated Read Time: 4-6 hours (full chapter)

Prerequisites:

Chapter 2: Platform Architecture
Chapter 8: Data Plane Overview
Familiarity with Python, FastAPI, and LLM concepts

Related Chapters:

Ch. 8: Data Plane Services
Ch. 10: Query Engine
Ch. 14: Ontology Service

What You Will Learn

This chapter is divided into eight major sections, each covering a distinct subsystem of the AI Service. The sections are organized from foundational infrastructure through to API reference documentation.

Section	Description	Audience
Service Architecture	Module organization, FastAPI application lifecycle, middleware stack, configuration, and deployment topology	Architects, Platform Engineers
Agent System	Agent orchestration, multi-agent teams, memory, guardrails, approval workflows, studio, drift detection, evaluation, and runtime lifecycle	AI/ML Engineers, Backend Developers
Text-to-SQL Pipeline	NL parsing, schema extraction, SQL generation, validation, auto-correction, query decomposition, and benchmarking	Data Engineers, AI/ML Engineers
BI Analytics Platform	Chat sessions, dashboards, alerts, export, filters, scheduling, sharing, visualization, and streaming	Data Analysts, Frontend Developers
Context Graph	Semantic search, graph traversal, agent thinking traces, ontology management, causal inference, and semantic SQL	Knowledge Engineers, AI/ML Engineers
LLM Infrastructure	Multi-provider router, response cache, context management, validation, MCP integration, and performance monitoring	Platform Engineers, AI/ML Engineers
ML Integration	Model training, hyperparameter tuning, feature engineering, model serving, registry, and experiment tracking	ML Engineers
Integrations	Kafka event streaming, WebSocket communication, feedback/RLHF, data lineage, dbt, personalization, and copilot features	Backend Developers, Platform Engineers
API Reference	Complete endpoint documentation for all REST, WebSocket, and GraphQL APIs	All Developers

12.1AI Service at a Glance

The AI Service is a Python 3.11+ / FastAPI application that serves as the conversational analytics brain of the MATIH platform. It receives natural language questions from frontend clients (BI Workbench, Agentic Workbench), routes them through specialized agents, generates SQL, retrieves relevant context via RAG, and returns structured responses with data, visualizations, and explanations.

Architecture Diagram

                    +---------------------------+
                    |     Frontend Clients       |
                    | (BI Workbench, Agentic UI) |
                    +------------+--------------+
                                 |
                    +------------v--------------+
                    |      AI Service           |
                    |     (Port 8000)           |
                    |   7 Feature-Flagged       |
                    |       Modules             |
                    +------------+--------------+
                                 |
     +--------+--------+--------+--------+--------+--------+
     |        |        |        |        |        |        |
  +--v---+ +--v---+ +--v---+ +--v----+ +-v-----+ +-v----+ +-v-------+
  | Core | | BI   | | ML   | | Data  | |Context| |Enter-| |Supple-  |
  |Module| |Platf.| |Platf.| |Platf. | |Graph  | |prise | |mentary  |
  +--+---+ +--+---+ +--+---+ +--+----+ +--+----+ +--+---+ +--+------+
     |        |        |        |          |         |        |
  +--v--------v--------v--------v----------v---------v--------v------+
  |                  Shared Infrastructure                            |
  |  LLM Router | RAG | Vector Store | Sessions | Cache | Kafka     |
  +--+-------+--------+---------+----------+----------+-------------+
     |       |        |         |          |          |
  +--v--+ +-v------+ +-v-----+ +-v------+ +-v------+ +-v------+
  |LLM  | |Qdrant  | |Query  | |Redis / | |Kafka   | |Dgraph  |
  |APIs  | |Vectors | |Engine | |Postgres| |Broker  | |Graph   |
  +------+ +--------+ +-------+ +--------+ +--------+ +--------+

Key Numbers

Metric	Value
Technology	Python 3.11+, FastAPI 0.100+, uvicorn
Service Port	8000
Source Files	984 Python files
Feature Modules	7 (Core, BI Platform, ML Platform, Data Platform, Context Graph, Enterprise, Supplementary)
LLM Providers	OpenAI, Anthropic, Azure OpenAI, Vertex AI, Bedrock, vLLM
Agent Types	80+ (Conversational, Specialist, Supervisor, Persona, Tool-specific)
Routing Strategies	Cost-optimized, Latency-optimized, Quality-optimized, Round-robin, Fallback
Streaming Protocols	WebSocket, Server-Sent Events (SSE)
Vector Store	Qdrant (gRPC/HTTP)
Session Backends	Redis (primary), PostgreSQL (persistent), Memory (fallback)
Kafka Topics	9 topics (feedback, learning signals, insights, agent thinking, decisions, traces, metrics)

12.2Module System

The AI Service uses a modular architecture where each module is a self-contained unit that registers its own routers and initializes its own services. Modules can be individually enabled or disabled via environment variables.

# Module feature flags from src/main.py
MODULE_FLAGS = {
    "core": os.environ.get("MODULE_CORE_ENABLED", "true").lower() == "true",
    "bi_platform": os.environ.get("MODULE_BI_ENABLED", "true").lower() == "true",
    "ml_platform": os.environ.get("MODULE_ML_ENABLED", "true").lower() == "true",
    "data_platform": os.environ.get("MODULE_DATA_ENABLED", "true").lower() == "true",
    "context_graph": os.environ.get("MODULE_CONTEXT_GRAPH_ENABLED", "true").lower() == "true",
    "enterprise": os.environ.get("MODULE_ENTERPRISE_ENABLED", "true").lower() == "true",
    "supplementary": os.environ.get("MODULE_SUPPLEMENTARY_ENABLED", "true").lower() == "true",
}

Module	Flag	Key Capabilities
Core	`MODULE_CORE_ENABLED`	Agents, LLM Router, Cache, MCP, Studio, Evaluation, Quality Metrics, Guardrails, Tools, Runtime Lifecycle
BI Platform	`MODULE_BI_ENABLED`	Analytics Orchestrator, Dashboards, Alerts, Export, Filters, Scheduling, Sharing, Semantic Models
ML Platform	`MODULE_ML_ENABLED`	Feature Store, Model Registry, Training, Serving, Tracking, Governance
Data Platform	`MODULE_DATA_ENABLED`	Catalog Integration, Pipeline Management, Data Quality
Context Graph	`MODULE_CONTEXT_GRAPH_ENABLED`	Knowledge Graph, Ontology, Semantic SQL, SHACL, Agent Thinking, Kafka Streaming
Enterprise	`MODULE_ENTERPRISE_ENABLED`	Billing, Compliance, Audit
Supplementary	`MODULE_SUPPLEMENTARY_ENABLED`	Search, Recommendations, FDME, Workflows, Feedback, Lineage, Personalization

12.3Key Source Files

The AI Service implementation is organized under data-plane/ai-service/src/:

Path	Purpose
`main.py`	FastAPI application entry point with module system
`config/settings.py`	Pydantic Settings with 60+ configuration parameters
`agents/orchestrator.py`	`AgentOrchestrator` and `MultiAgentOrchestrator`
`agents/studio/`	Agent creation, templates, preview, deployment
`agents/runtime/`	Agent platform feature flags, lifecycle, beta program, production rollout, GA readiness
`sql_generation/generator.py`	`SQLGenerator` pipeline
`sql_generation/validator.py`	`SQLValidator` with sqlglot/sqlparse
`bi/agents/orchestrator.py`	`AnalyticsOrchestrator` state machine
`bi/api/dashboard_routes.py`	Dashboard CRUD endpoints
`llm/providers/router.py`	`LLMRouter` with multi-provider routing
`llm/cache/cache_service.py`	`ResponseCacheService` with tenant isolation
`context_graph/`	Knowledge graph, ontology, semantic SQL, agent thinking
`feedback/`	Feedback collection, RLHF, Kafka integration
`session/`	Redis-backed session management
`storage/pool.py`	asyncpg connection pool management

12.4Integration Points

The AI Service integrates with nearly every other component in the MATIH platform:

Integration	Protocol	Purpose
Query Engine	HTTP REST	SQL execution against Trino/ClickHouse
Catalog Service	HTTP REST	Schema metadata discovery
Semantic Layer	HTTP REST	Business metric definitions
Ontology Service	HTTP REST	Entity relationships and semantic queries
Qdrant	gRPC/HTTP	Vector storage for schema embeddings and RAG
Redis	Redis protocol	Session storage, caching, pub/sub
PostgreSQL	asyncpg	Persistent session, feedback, and metrics storage
Kafka	aiokafka	Event streaming for feedback, context graph, agent metrics
Dgraph	HTTP/GraphQL	Agent thinking trace persistence
OpenAI	HTTPS	GPT-4o inference
Anthropic	HTTPS	Claude inference (with extended thinking)
Azure OpenAI	HTTPS	Azure-hosted OpenAI models
Vertex AI	gRPC/HTTPS	Gemini model inference
AWS Bedrock	HTTPS	Bedrock model inference
vLLM	HTTP	Self-hosted model inference

12.5Getting Started

To work with the AI Service locally:

# Navigate to service directory
cd data-plane/ai-service
 
# Install dependencies
pip install -r requirements.txt
 
# Set required environment variables
export JWT_SECRET_KEY=dev-secret-key-minimum-32-characters
export DATABASE_URL=postgresql://matih:password@localhost:5432/matih_ai
export REDIS_HOST=localhost
export REDIS_PORT=6379
export OPENAI_API_KEY=sk-xxx
export QDRANT_URL=http://localhost:6333
export KAFKA_BOOTSTRAP_SERVERS=localhost:9092
 
# Start the service
uvicorn src.main:app --host 0.0.0.0 --port 8000 --reload

The service exposes interactive API documentation at:

Swagger UI: http://localhost:8000/docs
ReDoc: http://localhost:8000/redoc

Health Endpoints

# General health
curl http://localhost:8000/health
 
# Semantic Platform health
curl http://localhost:8000/health/semantic-platform
 
# Agent Platform health
curl http://localhost:8000/health/agent-platform
 
# Database Pool health
curl http://localhost:8000/health/database-pool

Disabling Modules for Development

To run a minimal configuration during local development:

# Disable non-essential modules
export MODULE_ML_ENABLED=false
export MODULE_DATA_ENABLED=false
export MODULE_ENTERPRISE_ENABLED=false
export MODULE_SUPPLEMENTARY_ENABLED=false
 
# Run with only Core, BI Platform, and Context Graph
uvicorn src.main:app --host 0.0.0.0 --port 8000 --reload

Temporal Service Architecture