Secret Management

The MATIH Platform enforces a strict zero-trust policy for secrets: no passwords, API keys, tokens, or encryption keys are ever stored in source code, Helm values files, or container images. All secrets are injected at runtime through Kubernetes Secrets, managed by the External Secrets Operator (ESO) for production environments, and created by automation scripts for development environments.

Secret Management Architecture

Environment	Secret Source	Management Tool
Local Development	Docker Compose `.env` files	`docker-compose.yml` environment variables
Dev Kubernetes	Kubernetes Secrets	`scripts/lib/k8s/dev-secrets.sh`
Staging / Production	Cloud Key Vault	External Secrets Operator (ESO)

The architecture ensures that secrets flow from a secure source into the runtime environment without ever being committed to version control.

Cloud Key Vault (Azure / AWS / GCP)
        |
        |-- External Secrets Operator syncs secrets
        v
Kubernetes Secret (in tenant namespace)
        |
        |-- Mounted as env vars or volumes
        v
Application Container

Kubernetes Secrets

How Services Consume Secrets

Services consume secrets through Kubernetes secretKeyRef in their environment variables. This is the only approved method for passing secrets to containers:

# Helm template: deployment.yaml
env:
  - name: DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: ai-service-db-credentials
        key: password
  - name: JWT_SECRET
    valueFrom:
      secretKeyRef:
        name: platform-jwt-secret
        key: secret
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: ai-service-llm-keys
        key: openai-api-key

Connection String Construction

Database connection strings are constructed from secret references using environment variable substitution. Passwords are never hardcoded in connection URIs:

# CORRECT: Password from secret reference
env:
  - name: DB_HOST
    value: "postgresql.matih-shared.svc.cluster.local"
  - name: DB_PORT
    value: "5432"
  - name: DB_NAME
    value: "matih_ai_service"
  - name: DB_USER
    valueFrom:
      secretKeyRef:
        name: ai-service-db-credentials
        key: username
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: ai-service-db-credentials
        key: password
  - name: DATABASE_URL
    value: "postgresql://$(DB_USER):$(DB_PASSWORD)@$(DB_HOST):$(DB_PORT)/$(DB_NAME)"

Volume-Mounted Secrets

For secrets that need to be consumed as files (TLS certificates, SSH keys), volume mounts are used:

volumes:
  - name: tls-certs
    secret:
      secretName: ai-service-tls
      items:
        - key: tls.crt
          path: server.crt
        - key: tls.key
          path: server.key
 
volumeMounts:
  - name: tls-certs
    mountPath: /etc/tls
    readOnly: true

Development Secrets

In development environments, secrets are created by the dev-secrets.sh script. This script generates random passwords and creates Kubernetes Secrets without requiring any external key vault.

Using dev-secrets.sh

# Create all development secrets
./scripts/lib/k8s/dev-secrets.sh
 
# The script creates secrets like:
# - platform-jwt-secret
# - ai-service-db-credentials
# - ai-service-llm-keys
# - mlflow-s3-credentials
# - kafka-credentials
# - redis-credentials

Development Secret Pattern

The dev-secrets.sh script follows a consistent pattern:

Generate a random password or use a known development value.
Create a Kubernetes Secret in the target namespace.
The Helm chart references the secret via existingSecret and secretKeyRef.

# values-dev.yaml (CORRECT)
database:
  existingSecret: ai-service-db-credentials
  usernameKey: username
  passwordKey: password
 
# values-dev.yaml (WRONG -- never do this)
# database:
#   username: matih
#   password: dev-password-123

External Secrets Operator (ESO)

For staging and production environments, the External Secrets Operator (ESO) synchronizes secrets from cloud key vaults into Kubernetes Secrets automatically.

ESO Architecture

Azure Key Vault / AWS Secrets Manager / GCP Secret Manager
        |
        |  (ESO polls for changes)
        v
ExternalSecret Custom Resource (in Kubernetes)
        |
        |  (ESO creates/updates)
        v
Kubernetes Secret (standard K8s secret)
        |
        |  (Pod mounts via secretKeyRef)
        v
Application Container

ExternalSecret Resource

An ExternalSecret resource defines how to map a cloud secret to a Kubernetes Secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: ai-service-db-credentials
  namespace: tenant-acme-corp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-key-vault
    kind: ClusterSecretStore
  target:
    name: ai-service-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: acme-corp-ai-service-db-username
    - secretKey: password
      remoteRef:
        key: acme-corp-ai-service-db-password

ClusterSecretStore Configuration

The ClusterSecretStore defines the connection to the cloud key vault:

Azure Key Vault:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: azure-key-vault
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://matih-prod-vault.vault.azure.net"
      serviceAccountRef:
        name: external-secrets-sa
        namespace: matih-system

AWS Secrets Manager:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: matih-system

GCP Secret Manager:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-manager
spec:
  provider:
    gcpsm:
      projectID: matih-production
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: matih-prod-cluster
          serviceAccountRef:
            name: external-secrets-sa
            namespace: matih-system

Per-Tenant Secret Isolation

Each tenant's secrets are stored with tenant-specific key names in the cloud vault:

Azure Key Vault
├── acme-corp-db-password
├── acme-corp-openai-api-key
├── acme-corp-jwt-secret
├── globex-db-password
├── globex-openai-api-key
├── globex-jwt-secret
└── platform-master-key

ESO creates separate Kubernetes Secrets in each tenant namespace, ensuring that secrets for one tenant are never accessible to another.

Secret Rotation

Secret rotation is a critical operational practice. MATIH supports both manual and automated rotation.

Rotation Workflow

1. Generate new secret value in cloud vault
        |
2. ESO detects change (within refreshInterval)
        |
3. ESO updates Kubernetes Secret
        |
4. Application detects secret change
        |
5. Application switches to new secret
        |
6. Old secret revoked after grace period

Database Password Rotation

# 1. Update password in Azure Key Vault
az keyvault secret set \
  --vault-name matih-prod-vault \
  --name acme-corp-db-password \
  --value "new-secure-password"
 
# 2. ESO automatically syncs within the refresh interval (default: 1h)
# 3. Rolling restart picks up new secret
# kubectl rollout restart deployment/ai-service -n tenant-acme-corp

API Key Rotation

The API key rotation feature described in the Authentication section provides zero-downtime rotation with a configurable grace period where both old and new keys are valid.

JWT Secret Rotation

JWT secret rotation requires careful coordination:

Deploy the new secret alongside the old one.
New tokens are signed with the new secret.
Old tokens remain valid until they expire (max 15 minutes for access tokens).
After the maximum token lifetime, remove the old secret.

Secret Categories

Category	Examples	Rotation Frequency
Database credentials	PostgreSQL passwords, connection strings	90 days
API keys	OpenAI API keys, third-party service keys	Per vendor policy
JWT signing secrets	HMAC keys for token signing	180 days
Encryption keys	AES-256 keys for data encryption	365 days
TLS certificates	Service TLS certificates	Auto-renewed by cert-manager
Service tokens	Inter-service authentication	Auto-generated (5-min lifetime)

Secret Governance

Audit Trail

All secret access is logged in the audit trail:

Event	Logged Information
Secret created	Secret name, namespace, creator, timestamp
Secret accessed	Secret name, accessor service, timestamp
Secret rotated	Secret name, rotation reason, old/new version
Secret deleted	Secret name, deleter, reason, timestamp

Secret Scanning

The CI/CD pipeline includes secret scanning to prevent accidental commits:

Pre-commit hooks scan staged files for credential patterns
Pull request checks scan diffs for secrets
Repository scanning detects historical secret leaks

Next Steps

Continue to Data Encryption to understand how data is encrypted at rest and in transit, including the per-tenant key management system.

Tenant Isolation Data Encryption