Log Aggregation

MATIH uses Promtail (or Fluent-bit as an alternative) to collect logs from all Kubernetes pods and ship them to Loki. The log collector runs as a DaemonSet on every node, reads container log files, applies label enrichment, and forwards structured log entries.

Promtail Configuration

Promtail is deployed as part of the loki-stack Helm chart:

promtail:
  enabled: true
  config:
    clients:
      - url: http://loki:3100/loki/api/v1/push
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
          - source_labels: [__meta_kubernetes_pod_container_name]
            target_label: container
          - source_labels: [__meta_kubernetes_pod_label_app]
            target_label: service

Fluent-bit Alternative

For environments that prefer Fluent-bit:

fluent-bit:
  config:
    inputs: |
      [INPUT]
          Name              tail
          Tag               kube.*
          Path              /var/log/containers/*.log
          Parser            docker
          Refresh_Interval  10
          Mem_Buf_Limit     5MB
 
    filters: |
      [FILTER]
          Name                kubernetes
          Match               kube.*
          Merge_Log           On
          Keep_Log            Off
 
    outputs: |
      [OUTPUT]
          Name                loki
          Match               kube.*
          Host                loki
          Port                3100
          Labels              namespace=$kubernetes['namespace_name'],pod=$kubernetes['pod_name']

Label Enrichment

The log collector enriches log entries with Kubernetes metadata:

Label	Source	Description
`namespace`	Pod metadata	Kubernetes namespace
`pod`	Pod metadata	Pod name
`container`	Pod metadata	Container name
`service`	Pod label `app`	Service name
`node`	Node metadata	Node name

Log Pipeline

Pod stdout/stderr --> Container runtime log file
                          |
                    Promtail/Fluent-bit (DaemonSet)
                          |
                    Label enrichment
                          |
                    Loki push API
                          |
                    Loki storage
                          |
                    Grafana (query)

Multi-Line Log Handling

Stack traces and multi-line exceptions are combined into single log entries:

pipeline_stages:
  - multiline:
      firstline: '^\d{4}-\d{2}-\d{2}'
      max_lines: 128
      max_wait_time: 3s

Resource Limits

Component	CPU Request	CPU Limit	Memory Request	Memory Limit
Promtail	100m	200m	128Mi	256Mi
Fluent-bit	50m	100m	64Mi	128Mi

Filtering

To reduce log volume, exclude noisy logs at the collector level:

pipeline_stages:
  - drop:
      expression: "health_check|readiness_probe"
  - drop:
      source: level
      value: debug

Log Querying (LogQL)Alerting