Data Plane Namespace

The matih-data-plane namespace hosts all data processing, AI/ML, and analytics services alongside the data infrastructure components. This is the heaviest namespace in terms of resource consumption, with dedicated resource quotas and limit ranges.

Services Deployed

Service	Tech	Port	Replicas	Purpose
query-engine	Java	8080	3	SQL query routing and federation
catalog-service	Java	8086	2	Data catalog (OpenMetadata integration)
pipeline-service	Java	8092	2	ETL pipeline orchestration
semantic-layer	Java	8086	2	Semantic model management
bi-service	Java	8084	2	Dashboard and visualization
ai-service	Python	8000	2	NLP-to-SQL, conversational AI
ml-service	Python	8000	1	ML training and serving
data-quality-service	Python	8000	2	Data profiling and quality rules
data-plane-agent	Java	8085	1	Control plane communication
render-service	Node.js	8098	2	Chart/PDF rendering
ops-agent-service	Python	8080	2	Infrastructure optimization

Resource Quotas

The data plane enforces resource quotas to prevent runaway workloads:

# From matih-data-plane/templates/resource-quotas.yaml
resourceQuotas:
  enabled: true
  requests:
    cpu: "80"
    memory: "160Gi"
  limits:
    cpu: "160"
    memory: "320Gi"
  pods: 300
  storage:
    persistentVolumeClaims: 50
    requestsStorage: "500Gi"

Rendered Kubernetes resource:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: matih-data-plane-quota
  namespace: matih-data-plane
spec:
  hard:
    requests.cpu: "80"
    requests.memory: "160Gi"
    limits.cpu: "160"
    limits.memory: "320Gi"
    pods: "300"
    services: "100"
    secrets: "200"
    configmaps: "150"
    persistentvolumeclaims: "50"
    services.nodeports: "10"
    services.loadbalancers: "5"
    requests.storage: "500Gi"

Limit Ranges

Default resource limits prevent pods from consuming excessive resources:

# From matih-data-plane/values.yaml
limitRanges:
  enabled: true
  default:
    cpu: "500m"
    memory: "512Mi"
  defaultRequest:
    cpu: "100m"
    memory: "128Mi"
  max:
    cpu: "4"
    memory: "8Gi"

Data Infrastructure

The data plane namespace also hosts these infrastructure components:

Component	Type	Purpose
PostgreSQL	StatefulSet	Relational storage for all services
Redis	StatefulSet	Caching, sessions, pub/sub
Strimzi Kafka	Strimzi CRD	Event streaming, domain events
Trino	Deployment	Federated SQL engine
MinIO	StatefulSet	S3-compatible object storage
Qdrant	StatefulSet	Vector database for AI embeddings
ClickHouse	StatefulSet	OLAP analytics
Polaris	Deployment	Iceberg REST catalog
OpenMetadata	Deployment	Data catalog and governance

Node Selection

All data plane workloads target dedicated nodes:

nodeSelector:
  agentpool: dataplane
 
tolerations:
  - key: "matih.ai/data-plane"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

Global Configuration

Services share global configuration for database, Redis, Kafka, and storage:

global:
  namespace: matih-data-plane
  database:
    host: "postgresql.matih-data-plane.svc.cluster.local"
    port: 5432
  redis:
    host: "redis-master.matih-data-plane.svc.cluster.local"
    port: 6379
  kafka:
    bootstrapServers: "strimzi-kafka-kafka-bootstrap.matih-data-plane.svc.cluster.local:9093"
    securityProtocol: SSL
  trino:
    url: "jdbc:trino://trino.matih-data-plane.svc.cluster.local:8080"
  storage:
    type: "minio"
    s3:
      endpoint: "http://minio.matih-data-plane.svc.cluster.local:9000"

Control Plane Tenant Namespaces