Specialized Agents

Production - Integration agents for Airflow, Polaris, OpenMetadata, Ray, and Spark

Specialized Agents provide deep integrations with specific data platform components. Each agent understands the API, data model, and operational patterns of its target system.

12.2.14.1Available Specialized Agents

Agent	Target System	Capabilities
Airflow Agent	Apache Airflow	DAG management, trigger runs, check status, view logs
Polaris Agent	Apache Polaris	Catalog management, table operations, namespace management
OpenMetadata Agent	OpenMetadata	Metadata discovery, lineage queries, quality checks
Ray Agent	Ray Cluster	Job submission, cluster status, resource management
Spark Agent	Apache Spark	SparkSQL execution, job management, resource monitoring

Service Discovery

Specialized agents connect to their target systems via service discovery URLs configured in src/config/settings.py:

airflow_url: str = "http://airflow-api-server.matih-data-plane.svc.cluster.local:8080/api/v1"
polaris_url: str = "http://polaris.matih-catalog.svc.cluster.local:8181"
openmetadata_url: str = "http://openmetadata.matih-catalog.svc.cluster.local:8585"
ray_head_url: str = "http://ray-head.matih-data-plane.svc.cluster.local:8265"
spark_connect_url: str = "http://spark-connect.matih-data-plane.svc.cluster.local:15002"

12.2.14.2Airflow Agent Example

# Trigger an Airflow DAG via the agent
curl -X POST http://localhost:8000/api/v1/agents/process \
  -H "Content-Type: application/json" \
  -H "X-Tenant-ID: acme-corp" \
  -d '{
    "agent_id": "airflow-agent",
    "session_id": "session-123",
    "message": "Trigger the daily_sales_etl DAG with date=2025-01-15"
  }'

The Airflow agent translates natural language into Airflow API calls:

{
  "content": "I have triggered the daily_sales_etl DAG for 2025-01-15. The DAG run ID is run-abc-123.",
  "tool_calls": [
    {
      "name": "trigger_dag",
      "arguments": {
        "dag_id": "daily_sales_etl",
        "conf": {"date": "2025-01-15"}
      }
    }
  ],
  "tool_results": [
    {
      "name": "trigger_dag",
      "result": {"dag_run_id": "run-abc-123", "state": "queued"}
    }
  ]
}

Persona Agents Tool Management