Data Migration

When tenants upgrade from a shared cluster to dedicated infrastructure, their data must be migrated with zero data loss. The migration pipeline handles databases, object storage, message queues, and cache state.

Migration Components

Database Migration

Schema Isolation in Shared Clusters

In shared clusters, each tenant's data is isolated by schema or row-level security:

-- Shared cluster: tenant data isolated by tenant_id column
SELECT * FROM shared_schema.queries WHERE tenant_id = 'acme-tenant-id';
 
-- Dedicated cluster: full schema ownership
SELECT * FROM public.queries;

Migration Steps

1. Pre-migration snapshot: Create a point-in-time backup of tenant data
2. Schema export: Export all tenant-scoped tables using pg_dump with row filters
3. Schema creation: Create the database schema on the dedicated instance
4. Data import: Import using pg_restore with parallel workers
5. Verification: Row count comparison and checksum validation
6. Cutover: Update tenant connection string in config store

Object Storage Migration

Files and artifacts are migrated between storage accounts:

Shared: s3://matih-shared-data/tenants/acme/**
  --> Dedicated: s3://acme-dedicated-data/**

The migration service:

1. Lists all objects with the tenant prefix
2. Copies objects in parallel (configurable concurrency)
3. Verifies checksums (MD5/SHA-256) for each object
4. Updates storage references in the database

Migration Verification

After data migration completes, the system runs automated verification:

Rollback Procedure

If migration verification fails, the system rolls back:

1. Keep shared cluster data intact (never deleted until verification passes)
2. Drop dedicated database or mark as failed
3. Delete migrated objects from dedicated storage
4. Revert tenant configuration to point to shared cluster
5. Log failure details in the migration audit trail

Source Files