Pipeline Service Chart

The Pipeline Service orchestrates ETL pipelines with Airflow integration, dbt transformation support, and Spark job management.

Chart Configuration

pipeline-service:
  enabled: true
  replicaCount: 2
 
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "1"
      memory: "2Gi"
 
  config:
    airflow:
      baseUrl: "http://airflow-api-server:8080"
    dbt:
      enabled: true
      version: "1.7"
    spark:
      enabled: true
      masterUrl: "spark://spark-master:7077"

Pipeline Backends

Backend	Purpose	Connection
Airflow	DAG orchestration, scheduling	REST API on port 8080
dbt	SQL transformations	CLI / API
Spark	Distributed data processing	Spark Master on port 7077

Catalog Service ML Service