Chapter 11: Pipelines and Data Engineering

The Pipelines and Data Engineering layer provides the data movement, transformation, and orchestration capabilities of the MATIH platform. It encompasses the Pipeline Service (a unified FastAPI application), integrations with Apache Airflow for DAG orchestration, Apache Spark v4.1.1 for distributed processing, Apache Flink for real-time streaming, and Temporal for workflow orchestration. Pre-built pipeline templates accelerate development for common data engineering patterns across industries.

What You Will Learn

By the end of this chapter, you will understand:

• The Pipeline Service architecture, its unified API for managing pipelines across multiple execution engines, and the operator framework for data transformations
• Airflow integration for DAG orchestration, custom operators for MATIH workloads, DAG synchronization, and Airflow API management
• Spark integration with Spark v4.1.1 and Spark Connect for distributed data processing, Iceberg table operations, and Spark Operator on Kubernetes
• Flink streaming for CDC pipelines, real-time analytics aggregation, and the four production Flink SQL jobs
• Temporal workflows for long-running data pipeline orchestration, activity definitions, retry policies, and workflow monitoring
• Pipeline templates organized by industry vertical, providing pre-built transformation patterns for rapid pipeline development
• The complete API surface for pipeline management, monitoring, scheduling, and execution

Chapter Structure

Pipelines at a Glance

The pipeline layer supports four execution engines unified behind a single API:

                      +-------------------+
                      |  Pipeline Service |
                      |  (Python/FastAPI) |
                      |  Port 8000        |
                      +--------+----------+
                               |
              +----------------+----------------+----------------+
              |                |                |                |
     +--------v------+ +------v-------+ +------v-------+ +-----v-------+
     | Apache Airflow| | Apache Spark | | Apache Flink | | Temporal    |
     | (DAG Orch)    | | v4.1.1       | | (Streaming)  | | (Workflows) |
     +---------------+ +--------------+ +--------------+ +-------------+
              |                |                |                |
     +--------v------+ +------v-------+ +------v-------+ +-----v-------+
     | Scheduled ETL | | Distributed  | | CDC / Real-  | | Long-running|
     | Batch jobs    | | Processing   | | time Agg     | | Orchestrtion|
     +---------------+ +--------------+ +--------------+ +-------------+

Key Numbers

Key Source Files

Design Principles

1. Engine agnosticism. The Pipeline Service provides a unified interface for pipeline definition regardless of whether execution happens on Airflow, Spark, Flink, or Temporal.

2. Template-driven development. Pre-built templates for common patterns reduce pipeline development from days to hours.

3. Lineage by default. Every pipeline execution emits OpenLineage events for automatic lineage tracking.

4. Quality gates. Pipelines integrate with the Data Quality Service to validate data at critical transformation points.

5. Cost awareness. Pipeline execution includes cost estimation and tracking for resource optimization.

6. Observability everywhere. Every pipeline execution emits metrics, logs, and traces for complete visibility.

Execution Engine Selection

The Pipeline Service selects the appropriate execution engine based on pipeline characteristics:

How This Chapter Connects

The Pipeline Service integrates with several platform services:

• The Data Catalog (Chapter 10) receives lineage events from pipeline executions and provides schema metadata for transformations
• The Data Quality Service (Chapter 10) validates data at quality gates within pipeline stages
• The Query Engine (Chapter 9) executes SQL transformations and provides data access for pipeline reads
• The AI Service (Chapter 12) consumes the analytics produced by Flink streaming jobs

Begin with the Pipeline Service section to understand the unified architecture.