Rollback Procedures

When provisioning fails and retries are exhausted, the system rolls back completed steps in reverse order. The ProvisioningController provides manual rollback and retry endpoints.

Automatic Rollback

When a step fails after exhausting all retries (maxRetries=3):

1. Tenant status is set to FAILED
2. Error message is stored in tenant.provisioningError
3. Rollback is initiated for all completed steps in reverse order

public void rollbackProvisioning(UUID tenantId) {
    List<ProvisioningStep> completedSteps =
            stepRepository.findCompletedStepsForRollback(tenantId);
 
    for (ProvisioningStep step : completedSteps) {
        try {
            stepExecutor.rollback(step);
            step.markRolledBack();
        } catch (Exception e) {
            step.markRollbackFailed(e.getMessage());
        }
        stepRepository.save(step);
    }
}

Manual Rollback

Platform administrators can trigger a rollback manually:

curl -X POST http://localhost:8082/api/v1/tenants/{tenantId}/provisioning/rollback \
  -H "Authorization: Bearer $TOKEN"

Returns 202 Accepted with the current provisioning status.

Retry Failed Provisioning

Failed provisioning can be retried, which resets failed steps and re-executes:

curl -X POST http://localhost:8082/api/v1/tenants/{tenantId}/provisioning/retry \
  -H "Authorization: Bearer $TOKEN"

The retry endpoint:

1. Validates that there are failed steps
2. Resets failed steps (increments retryCount, resets status to PENDING)
3. Re-executes provisioning from the first pending step

Reprovisioning

For a complete fresh start, the reprovision endpoint first rolls back all resources, then starts a new provisioning:

curl -X POST http://localhost:8082/api/v1/tenants/{tenantId}/provisioning/reprovision \
  -H "Authorization: Bearer $TOKEN"

Real-Time Status via SSE

The provisioning controller provides a Server-Sent Events stream for real-time provisioning updates:

curl -N http://localhost:8082/api/v1/tenants/{tenantId}/provisioning/status/stream \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: text/event-stream"

The SSE emitter sends status updates every 2 seconds and automatically completes when provisioning finishes (COMPLETED or FAILED). The emitter has a 30-minute timeout.

Source Files