Data Infrastructure
The MATIH Platform uses nine distinct data store technologies, each selected for specific workload characteristics. This section documents each data store, its role in the platform, and the services that depend on it.
Data Store Overview
| Technology | Version | Category | Primary Purpose |
|---|---|---|---|
| PostgreSQL | 16 | Relational | Primary transactional database for all services |
| Redis | 7 | Key-value | Caching, sessions, pub/sub, rate limiting |
| Apache Kafka | Strimzi 0.38+ | Event streaming | Asynchronous communication, event sourcing |
| Elasticsearch | 8.11 | Search | Full-text search, audit log indexing |
| MinIO | Latest | Object storage | S3-compatible artifact and file storage |
| Trino | Latest | Query federation | Federated SQL across multiple data sources |
| ClickHouse | Latest | OLAP | Fast analytical queries on large datasets |
| Qdrant | Latest | Vector database | RAG embeddings, semantic search |
| Neo4j / Dgraph | 5.x | Graph database | Knowledge graphs, data lineage |
PostgreSQL
PostgreSQL 16 is the primary relational database for every service in the platform.
| Aspect | Details |
|---|---|
| Multi-tenancy | Schema-per-tenant via Hibernate multi-tenancy |
| Services using it | All 24 services |
| Connection pooling | HikariCP with per-service pool configuration |
| Deployment | Kubernetes StatefulSet via Helm |
| Backup | pg_dump with schema filter for per-tenant backup |
Each service has its own database within the PostgreSQL instance, and each tenant has its own schema within the service database.
Redis
Redis 7 provides low-latency caching, session management, and real-time messaging.
| Aspect | Details |
|---|---|
| Caching | Tenant-aware key namespacing via TenantAwareCacheManager |
| Sessions | JWT session storage with configurable TTL |
| Pub/Sub | Configuration change propagation, AI response streaming |
| Rate limiting | Per-tenant API rate limiting counters |
| Services using it | 16 services |
Apache Kafka
Kafka provides durable event streaming for asynchronous communication between services.
| Aspect | Details |
|---|---|
| Deployment | Strimzi Kafka Operator on Kubernetes |
| Brokers | 1 (dev) / 3 (production) |
| Replication | 1 (dev) / 3 (production) with 2 min in-sync replicas |
| Partitioning | Tenant ID as partition key |
| Retention | 7 days (dev) / 30 days (production) |
| Compression | Snappy |
| Services using it | 10 services |
Elasticsearch
Elasticsearch provides full-text search and log indexing.
| Aspect | Details |
|---|---|
| Version | 8.11 |
| Use cases | Audit log search, ontology search, analytics |
| Services using it | Audit Service, Observability API, Ontology Service |
MinIO
MinIO provides S3-compatible object storage for artifacts and files.
| Aspect | Details |
|---|---|
| Use cases | ML model artifacts, pipeline outputs, exported reports, dashboard assets |
| API compatibility | Amazon S3 API |
| Authentication | Access key / secret key via Kubernetes secrets |
| Services using it | ML Service, Pipeline Service, Render Service |
Selection Rationale
| Technology | Why Selected Over Alternatives |
|---|---|
| PostgreSQL | Mature, excellent Hibernate support, schema-per-tenant multi-tenancy |
| Redis | Best-in-class latency for caching, built-in pub/sub |
| Kafka (Strimzi) | Kubernetes-native operator, durable event streaming, wide ecosystem |
| Elasticsearch | Industry standard for full-text search and log analytics |
| MinIO | S3-compatible without cloud vendor lock-in, Kubernetes-native |
Related Pages
- Data Stores (Architecture) -- Deep dive into data store architecture
- Compute Engines -- Trino, ClickHouse, Spark, Flink
- Backend Stack -- Service technology choices