Yash Verma / Cloud runbook

Roll back a bad release

A deployment went out and errors or latency went up. All playbooks.

  1. Confirm it is the release: compare the error rate before and after the deploy time.

    PromQL: HTTP error rate

    Calculates the percentage of 5xx responses over the last 5 minutes.

    sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100

    promql, errors, sre

  2. For a plain Deployment, list the revisions.

    View rollout history

    Checks the revision history of a deployment.

    kubectl rollout history deployment/<deployment-name>

    rollout, history

  3. Go back to the previous revision.

    Rollback a deployment

    Rolls back a deployment to the previous revision.

    kubectl rollout undo deployment/<deployment-name>

    rollback, undo, deployment

  4. Wait until the rollback has finished.

    Watch a rollout until it finishes

    Waits for a deployment rollout and exits non-zero if it fails.

    kubectl rollout status deployment/<deployment-name> -n <namespace>

    rollout, deploy

  5. For a Helm release, list its revisions instead.

    View release history

    Shows the revision history and status of a Helm release.

    helm history <release-name>

    helm, history, revisions

  6. Roll back to the last good revision.

    Rollback a release

    Rolls back a Helm deployment to a specific prior revision.

    helm rollback <release-name> <revision-number>

    helm, rollback, revert