Skip to content

Runbook: Rollback Deployment

When to use: A recent deployment caused an SLA breach, crash loops, or a regression in business metrics. This runbook covers ArgoCD-managed Helm deployments (all chakra-* services).

Step 1: Confirm the Deployment is the Cause

# Check when the last sync happened
argocd app history chakra-{service}

# Confirm the SLA alert timestamp aligns with the deployment window
# If the alert fired within 5 minutes of sync → deployment is suspected cause

If the alert preceded the deployment, the deployment is not the root cause — go back to SLA Breach Response.

Step 2: Roll Back via ArgoCD

# List recent revisions
argocd app history chakra-{service}

# Roll back to the previous successful revision
argocd app rollback chakra-{service} <PREVIOUS_REVISION_ID>

# Watch the rollout
kubectl rollout status deploy/{service} -n chakra-{service}

ArgoCD rollback syncs the cluster state to the previous Helm chart revision. It does not modify Git — the repo still contains the bad version.

Step 3: Verify Recovery

# Check pod health
kubectl get pods -n chakra-{service}

# Confirm no crash loops
kubectl describe deploy/{service} -n chakra-{service} | grep -A5 Conditions

# Check Grafana: SLA burn rate alert must resolve within 15 minutes

If pods are healthy but the burn rate alert is still firing after 15 minutes, the rollback may not have been the fix — see SLA Breach Response for other root cause categories.

Step 4: Prevent Re-Deployment

ArgoCD will try to re-sync to the current Git HEAD (the bad version). To prevent this until the fix is merged:

# Suspend auto-sync for the application
argocd app set chakra-{service} --sync-policy none

Re-enable auto-sync once the fix is merged and validated in staging:

argocd app set chakra-{service} --sync-policy automated --self-heal

Step 5: Post-Rollback Actions

  1. Create an incident ticket with: deployment SHA, alert timestamp, rollback timestamp, impact duration.
  2. Fix the bad change in a new PR. Do not push to main until the fix is tested.
  3. Re-enable ArgoCD auto-sync after the fix is merged.
  4. Update this runbook if a new rollback scenario was discovered.

Rollback During Migration Canary

If the rollback is for a canary-weight service (during active migration), also:

  1. Set the Kong canary weight to 0% for the affected service path.
  2. Confirm dual-write relay is still running (if applicable).
  3. Notify the migration team — the canary ramp is paused until the fix is validated.