Chapter 18: CI/CD and Build System

Developer Workflow
==================

  git push --> GitHub Actions CI
               |
               +--> build.sh --test-only (unit tests)
               +--> helm lint (chart validation)
               +--> terraform validate (IaC checks)
               +--> pre-deploy validation

  Merge to main --> CD Pipeline (cd-new.sh all dev)
                    |
                    +--> Stage 00: Terraform (Azure/AWS/GCP infra)
                    +--> Stage 01: Build Setup (Docker buildx, schema validation)
                    +--> Stage 02: Base Images (Java, Python, Node.js base images)
                    +--> Stage 03: Commons (shared libraries)
                    +--> Stage 04: Service Images (all service Docker images)
                    +--> Stage 05a: Control Plane Infrastructure (PostgreSQL, Redis, Kafka)
                    +--> Stage 05b: Data Plane Infrastructure (PostgreSQL, Redis, Kafka)
                    +--> Stage 06: Ingress Controller (NGINX)
                    +--> Stage 07: Control Plane Monitoring (Prometheus, Grafana)
                    +--> Stage 08: Control Plane Services (IAM, tenant, config, etc.)
                    +--> Stage 09: Control Plane Frontend (control-plane-ui)
                    +--> Stage 10: Data Plane Monitoring
                    +--> Stage 11: Compute Engines (Spark, Flink, Ray, Trino)
                    +--> Stage 12: Workflow Orchestration (Airflow)
                    +--> Stage 13: Data Catalogs (OpenMetadata)
                    +--> Stage 14: ML Infrastructure (KubeRay, MLflow)
                    +--> Stage 15: AI Infrastructure (vLLM, Ollama)
                    +--> Stage 16: Data Plane Services (ai-service, ml-service, etc.)
                    +--> Stage 17: Data Plane Frontend (workbenches)
                    +--> Stage 18: Validation (health checks, smoke tests)
                    |
                    +--> Auto-rollback on failure
                    +--> Build nodepool cleanup

# Version control
RELEASE_VERSION=1.2.3          # Semantic version for deployment
IMAGE_TAG=sha-abc123           # Docker image tag (default: latest)
 
# Pipeline behavior
DRY_RUN=true                   # Preview without executing
ROLLBACK_ON_FAILURE=true       # Auto-rollback on stage failure (default: true)
SKIP_DEPENDENCY_CHECK=true     # Skip dependency verification
SKIP_AI_INFRA=true             # Mark AI infrastructure as optional
SKIP_SCHEMA_VALIDATION=true    # Skip schema validation stage
 
# Build configuration
BUILD_CLEANUP_ENABLED=true     # Enable build nodepool cleanup (default: true)
BUILD_CLEANUP_WAIT_TIMEOUT=180 # Timeout for nodepool scale-down
FULL_SCHEMA_VALIDATION=true    # Run full Hibernate validation
KUBECTL_TIMEOUT=10             # Timeout for kubectl commands in seconds
 
# Registry
ACR_NAME=matihacr              # Azure Container Registry name
REGISTRY=ghcr.io/matih         # Container registry URL