Loki Setup

Grafana Loki is the centralized log aggregation backend for MATIH. It stores structured log data from all services and provides LogQL-based querying through Grafana. Loki is optimized for cost-effective log storage by indexing only labels rather than full log content.

Installation

Loki is deployed using the Grafana Loki Helm chart:

helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
  --namespace matih-monitoring \
  --set loki.persistence.enabled=true \
  --set loki.persistence.size=50Gi \
  --set promtail.enabled=true

Configuration

loki:
  auth_enabled: false
  server:
    http_listen_port: 3100
  schema_config:
    configs:
      - from: "2025-01-01"
        store: boltdb-shipper
        object_store: filesystem
        schema: v11
        index:
          prefix: index_
          period: 24h
  storage_config:
    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-shipper-active
      cache_location: /data/loki/boltdb-shipper-cache
    filesystem:
      directory: /data/loki/chunks
  limits_config:
    retention_period: 720h    # 30 days
    max_query_series: 5000
    max_entries_limit_per_query: 10000

Label Strategy

Loki indexes labels for efficient querying. MATIH uses the following label set:

Label	Description	Example
`namespace`	Kubernetes namespace	`matih-data-plane`
`pod`	Pod name	`ai-service-7d8f9c4b6-x2k3m`
`container`	Container name	`ai-service`
`service`	Service name	`ai-service`
`level`	Log level	`info`, `error`

High-cardinality values like tenant_id, trace_id, and request_id are stored in the log line content rather than as labels to keep index size manageable.

Retention

Environment	Retention	Storage
Development	7 days	10Gi PVC
Staging	14 days	25Gi PVC
Production	30 days	100Gi PVC (or object storage)

Production Storage Backend

For production, Loki should use object storage instead of local filesystem:

storage_config:
  aws:
    s3: s3://region/bucket-name
    bucketnames: matih-loki-logs
  boltdb_shipper:
    shared_store: s3

Accessing Loki

Loki is accessed exclusively through Grafana's Explore view using LogQL queries. There is no standalone UI.

# Verify Loki is running
kubectl port-forward svc/loki 3100:3100 -n matih-monitoring
curl http://localhost:3100/ready

Structured Logging Log Querying (LogQL)