Distributed Tracing Overview

MATIH implements distributed tracing using OpenTelemetry for instrumentation and Tempo as the trace storage backend. Traces propagate across microservice boundaries using W3C Trace Context headers, enabling end-to-end request visibility from the API gateway through the control plane and data plane services.

Subsections

Page	Description
OpenTelemetry Setup	SDK installation, configuration, and auto-instrumentation
Jaeger Integration	Jaeger UI for trace visualization (alternative to Tempo)
Trace Correlation	Correlating traces with logs and metrics
Span Analysis	Analyzing span trees for performance bottlenecks

Architecture

+-------------------+     +-------------------+     +------------------+
| API Gateway       | --> | Control Plane     | --> | Data Plane       |
| (trace context)   |     | (trace context)   |     | (trace context)  |
+--------+----------+     +--------+----------+     +--------+---------+
         |                         |                          |
         +------------+------------+--------------------------+
                      |
              +-------v--------+
              |  OTel Collector |
              +-------+--------+
                      |
              +-------v--------+
              |     Tempo       |
              +-------+--------+
                      |
              +-------v--------+
              |    Grafana      |
              +----------------+

Trace Context Propagation

Traces are propagated between services using the W3C Trace Context standard:

Header	Description
`traceparent`	Trace ID, span ID, and trace flags
`tracestate`	Vendor-specific trace data
`X-Tenant-Id`	Tenant context (custom header)

Distributed Tracing OpenTelemetry Setup