Python Builds (pip/poetry)

The MATIH platform's Python services use FastAPI with either pip or Poetry for dependency management. The build system automatically detects the package manager by examining pyproject.toml.

Python Service Inventory

Service	Port	Package Manager	Description
ai-service	8000	pip	Conversational AI, agent orchestration
ml-service	8000	pip	ML model management and inference
data-quality-service	8000	pip	Data quality monitoring and rules
pipeline-service	8092	pip	Data pipeline orchestration
ontology-service	8101	pip	Ontology and knowledge graph management
ops-agent-service	8080	pip	Infrastructure optimization agent
ops-agent-service/infra-optimizer	-	pip	Sub-module for infrastructure optimization

Build Detection Logic

The build script uses a two-step detection process:

build_python_project() {
    local project_path="$1"
    cd "$project_path"
 
    # Step 1: Require pyproject.toml
    if [[ ! -f "pyproject.toml" ]]; then
        return  # Skip directories without pyproject.toml
    fi
 
    # Step 2: Detect Poetry vs pip
    if command -v poetry &> /dev/null && grep -q "poetry" pyproject.toml; then
        # Poetry project
        poetry install
    else
        # pip/hatch project - install with dev extras
        python3.11 -m pip install -e ".[dev]" --quiet
    fi
}

Python Version Selection

The build system prefers Python 3.11 but falls back gracefully:

# Try python3.11 first, fall back to python3
python3.11 -m pip install -e ".[dev]" --quiet 2>/dev/null || \
python3 -m pip install -e ".[dev]" --quiet 2>/dev/null

Commons Python Library

The commons-python package provides shared utilities across all Python services:

commons/commons-python/
  ├── pyproject.toml
  └── src/
      └── matih_commons/
          ├── auth/          # JWT validation, tenant context
          ├── config/        # Settings management
          ├── logging/       # Structured logging setup
          ├── metrics/       # Prometheus metrics helpers
          ├── middleware/     # FastAPI middleware
          └── models/        # Shared Pydantic models

Built and installed first so services can import it:

cd commons/commons-python
python3.11 -m pip install -e ".[dev]" --quiet

Test Execution

Python tests use pytest with verbose output:

run_python_tests() {
    local project_path="$1"
    cd "$project_path"
 
    if [[ ! -d "tests" ]]; then
        return  # Skip if no tests directory
    fi
 
    local python_cmd="python3.11"
    if ! command -v python3.11 &> /dev/null; then
        python_cmd="python3"
    fi
 
    # Run without coverage threshold to avoid false failures
    $python_cmd -m pytest tests/ -v --tb=short --no-cov
}

Key testing patterns:

Tests must live in a tests/ directory
Coverage thresholds are disabled during CI builds (--no-cov)
Short tracebacks (--tb=short) for readable output
Verbose mode (-v) for individual test visibility

Docker Images for Python Services

Python services use multi-stage Dockerfiles with base images:

FROM matih/base-python:latest AS builder
WORKDIR /app
COPY pyproject.toml .
RUN pip install --no-cache-dir -e ".[prod]"
 
FROM matih/base-python:latest
WORKDIR /app
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY src/ src/
EXPOSE 8000
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

AI Service Build Specifics

The ai-service is the most complex Python service, with additional dependencies for:

LLM integration: OpenAI, Anthropic, Azure OpenAI SDKs
Vector databases: ChromaDB, pgvector
Agent frameworks: Custom orchestrator with state machines
Context graph: Dgraph client, embedding models

Its build includes additional compilation steps for native extensions used by ML libraries.

Java Builds (Maven)TypeScript Builds (npm/Vite)