Notification Channels

Alertmanager routes alerts to different notification channels based on severity and category. MATIH integrates with PagerDuty for critical pages, Slack for operational notifications, email for non-urgent alerts, and webhooks for custom integrations.

Channel Configuration

Channel	Severity	Use Case
PagerDuty	critical	Immediate paging of on-call engineer
Slack (#matih-alerts)	all	General alert notifications
Slack (#matih-warnings)	warning	Non-critical warnings
Slack (#matih-provisioning)	provisioning	Provisioning-specific alerts
Email	info	Weekly digest, capacity reports
Webhook	all	Custom integrations, ticketing systems

PagerDuty Integration

receivers:
  - name: pagerduty-critical
    pagerduty_configs:
      - service_key_file: /etc/alertmanager/secrets/pagerduty-key
        severity: critical
        description: |
          Alert: {{ .GroupLabels.alertname }}
          {{ .CommonAnnotations.summary }}
        details:
          firing: "{{ .Alerts.Firing | len }}"
          description: "{{ .CommonAnnotations.description }}"
          runbook: "{{ .CommonAnnotations.runbook_url }}"

PagerDuty service keys are stored in Kubernetes secrets and mounted into the Alertmanager pod.

Slack Integration

receivers:
  - name: slack-warnings
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: "#matih-warnings"
        send_resolved: true
        title: '{{ .GroupLabels.alertname }}'
        text: |
          {{ .CommonAnnotations.description }}
          Runbook: {{ .CommonAnnotations.runbook_url }}
        color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'

Email Integration

receivers:
  - name: email-digest
    email_configs:
      - to: platform-team@company.com
        from: alerts@matih.ai
        smarthost: smtp.company.com:587
        auth_username_file: /etc/alertmanager/secrets/smtp-user
        auth_password_file: /etc/alertmanager/secrets/smtp-pass
        send_resolved: true

Webhook Integration

For custom integrations (e.g., creating Jira tickets):

receivers:
  - name: webhook-tickets
    webhook_configs:
      - url: https://automation.company.com/alerts
        send_resolved: true
        http_config:
          bearer_token_file: /etc/alertmanager/secrets/webhook-token

Template Customization

Alert templates can be customized for each channel:

templates:
  - /etc/alertmanager/templates/*.tmpl

Template files use Go templating syntax with access to alert data, labels, annotations, and status.

Testing Notifications

Test alert routing using amtool:

amtool alert add alertname=TestAlert severity=warning namespace=matih-test \
  --annotation.summary="Test alert for notification channel validation"

Verify the alert appears in the expected channel, then resolve it:

amtool silence add alertname=TestAlert --duration=1m

Alert Rules Library Incident Response