Drift Detection

Production - Data drift monitoring, statistical tests, alerting, and remediation

The Drift Detection system monitors agent performance and data distributions for changes that may indicate degraded model quality. It performs statistical tests on input distributions, output quality metrics, and response patterns, alerting operators when significant drift is detected.

12.2.9.1Drift Detection Architecture

Implemented in data-plane/ai-service/src/agents/drift_detection/:

Component	File	Purpose
`DriftDetectionService`	`drift_detection_service.py`	Core drift analysis engine
`DriftDetectionRoutes`	`drift_detection_routes.py`	REST API endpoints
`PostgresDriftStore`	`postgres_store.py`	Persistent drift metric storage

Detection Methods

Method	Type	Description
Kolmogorov-Smirnov	Statistical	Tests distribution shift in numeric features
Chi-squared	Statistical	Tests distribution shift in categorical features
Population Stability Index	Statistical	Measures overall distribution stability
Jensen-Shannon Divergence	Information-theoretic	Measures distribution similarity
Moving Average	Heuristic	Tracks rolling averages of quality metrics

12.2.9.2API Endpoints

# Run drift detection for an agent
curl -X POST http://localhost:8000/api/v1/drift-detection/analyze \
  -H "Content-Type: application/json" \
  -H "X-Tenant-ID: acme-corp" \
  -d '{
    "agent_id": "sql-agent-prod",
    "baseline_window_days": 30,
    "current_window_days": 7,
    "metrics": ["response_quality", "token_usage", "latency", "error_rate"]
  }'
 
# Get drift report
curl http://localhost:8000/api/v1/drift-detection/report/{agent_id}?tenant_id=acme-corp
 
# Configure drift alert thresholds
curl -X PUT http://localhost:8000/api/v1/drift-detection/thresholds \
  -H "Content-Type: application/json" \
  -H "X-Tenant-ID: acme-corp" \
  -d '{
    "agent_id": "sql-agent-prod",
    "quality_score_min": 0.75,
    "error_rate_max": 0.05,
    "latency_p95_max_ms": 5000
  }'

Drift Report

{
  "agent_id": "sql-agent-prod",
  "analysis_window": {"start": "2025-01-08", "end": "2025-01-15"},
  "baseline_window": {"start": "2024-12-15", "end": "2025-01-08"},
  "drift_detected": true,
  "metrics": {
    "response_quality": {
      "baseline_mean": 0.87,
      "current_mean": 0.79,
      "ks_statistic": 0.23,
      "p_value": 0.003,
      "drift_severity": "moderate"
    },
    "token_usage": {
      "baseline_mean": 1450,
      "current_mean": 1820,
      "drift_severity": "minor"
    }
  },
  "recommendations": [
    "Review recent prompt changes",
    "Check for schema changes in connected databases",
    "Consider retraining or prompt adjustment"
  ]
}

Approval Workflows (HITL)Hallucination Classifier