Secrets Management

Secrets management in the MATIH platform follows a zero-hardcoded-credentials policy. All sensitive values including database passwords, API keys, and TLS certificates are stored in Kubernetes Secrets, with production environments using the External Secrets Operator (ESO) to sync from cloud key vaults.

Secrets Architecture

Development:
  dev-secrets.sh --> kubectl create secret --> Kubernetes Secrets --> Pod env vars

Production:
  Cloud Key Vault --> External Secrets Operator --> Kubernetes Secrets --> Pod env vars

Secret Categories

Category	Examples	Storage	Rotation
Database credentials	PostgreSQL, Redis, Kafka passwords	Kubernetes Secret	Manual (dev), Automated (prod)
API keys	OpenAI, Anthropic, cloud provider keys	Kubernetes Secret	Manual
TLS certificates	Service TLS, ingress TLS	cert-manager Secret	Automated
Service tokens	JWT signing keys, inter-service tokens	Kubernetes Secret	Periodic
Object store	MinIO/S3 access keys	Kubernetes Secret	Manual

Development Secrets

In development, secrets are created using the dev-secrets.sh script:

# Create all development secrets
./scripts/lib/k8s/dev-secrets.sh

This script creates Kubernetes Secrets with development-safe values. Secrets are never committed to Git.

Production Secrets with External Secrets Operator

In production, ESO syncs secrets from cloud key vaults:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: ai-service-secrets
  namespace: matih-data-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-key-vault
    kind: ClusterSecretStore
  target:
    name: ai-service-secrets
    creationPolicy: Owner
  data:
    - secretKey: database-url
      remoteRef:
        key: ai-service-database-url
    - secretKey: openai-api-key
      remoteRef:
        key: openai-api-key
    - secretKey: jwt-secret-key
      remoteRef:
        key: jwt-secret-key

Secret References in Helm Charts

Services reference secrets through secretKeyRef in environment variables:

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: ai-service-secrets
        key: database-url
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: ai-service-secrets
        key: openai-api-key
  - name: REDIS_PASSWORD
    valueFrom:
      secretKeyRef:
        name: redis-credentials
        key: password

Credentials are never placed directly in values.yaml or values-dev.yaml files.

Supported Key Vault Backends

Provider	Service	ESO Provider
Azure	Azure Key Vault	`azure/keyvault`
AWS	AWS Secrets Manager	`aws/secretsmanager`
GCP	GCP Secret Manager	`gcp/secretmanager`
HashiCorp	Vault	`vault`

Secret Rotation

Method	Trigger	Downtime
ESO refresh	Automatic (refreshInterval)	None (env var update on next pod restart)
Rolling restart	Manual or automated after ESO refresh	Zero (rolling update)
cert-manager renewal	Automatic (30 days before expiry)	None

Security Best Practices

Practice	Implementation
Encryption at rest	Kubernetes etcd encryption enabled
Encryption in transit	TLS for all secret access
Least privilege	RBAC restricts secret access to owning namespace
Audit logging	All secret access logged in audit trail
No git commits	Pre-commit hook blocks `.env` files and known secret patterns
Rotation	Automated via ESO with configurable refresh interval

Troubleshooting

Issue	Symptom	Resolution
`CreateContainerConfigError`	Pod fails with "secret not found"	Create missing secret or check name
`ExternalSecret` not syncing	Secret not updated from vault	Check ESO logs and vault permissions
Permission denied on secret	RBAC error accessing secret	Verify ServiceAccount has secret read access
Stale credentials	Application using old password	Restart pods after secret rotation

Security Contexts cert-manager