Quality Metrics
Production - Quality metric calculation, scoring, trending, and threshold alerts
The Quality Metrics system calculates and tracks quality scores for agent responses over time. It provides real-time quality scoring, historical trending, threshold-based alerting, and detailed metric breakdowns.
12.2.12.1Metrics Architecture
Implemented in data-plane/ai-service/src/agents/quality_metrics/:
| Component | Purpose |
|---|---|
QualityMetricsService | Calculates and aggregates quality metrics |
QualityMetricsRoutes | REST API endpoints |
QualityMetricsStore | In-memory metric storage |
PostgresQualityMetricsStore | Persistent PostgreSQL storage |
Quality Dimensions
| Dimension | Weight | Description |
|---|---|---|
| Accuracy | 0.30 | Correctness of factual claims and data |
| Relevance | 0.25 | How well the response addresses the question |
| Completeness | 0.20 | Coverage of all aspects of the question |
| Clarity | 0.15 | Readability and coherence of the response |
| Safety | 0.10 | Absence of harmful, biased, or inappropriate content |
12.2.12.2API Endpoints
# Record quality metric
curl -X POST http://localhost:8000/api/v1/quality-metrics \
-H "Content-Type: application/json" \
-H "X-Tenant-ID: acme-corp" \
-d '{
"agent_id": "sql-agent-prod",
"session_id": "session-123",
"dimensions": {
"accuracy": 0.92,
"relevance": 0.88,
"completeness": 0.85,
"clarity": 0.90,
"safety": 1.0
}
}'
# Get quality trend
curl "http://localhost:8000/api/v1/quality-metrics/trend?tenant_id=acme-corp&agent_id=sql-agent-prod&window_days=30"
# Get quality summary
curl "http://localhost:8000/api/v1/quality-metrics/summary?tenant_id=acme-corp&agent_id=sql-agent-prod"Quality Summary Response
{
"agent_id": "sql-agent-prod",
"period": "2025-01-01 to 2025-01-15",
"overall_score": 0.87,
"dimensions": {
"accuracy": {"mean": 0.89, "p50": 0.91, "p5": 0.72},
"relevance": {"mean": 0.86, "p50": 0.88, "p5": 0.68},
"completeness": {"mean": 0.84, "p50": 0.86, "p5": 0.65},
"clarity": {"mean": 0.88, "p50": 0.90, "p5": 0.75},
"safety": {"mean": 0.99, "p50": 1.0, "p5": 0.95}
},
"total_evaluations": 1247,
"trend": "stable"
}