Skip to content

Runbook: Code rollback

Symptom

You are here because one of: - A newly deployed release is misbehaving (regressions, perf cliff, unexpected error spikes, OOM trend — see ./oom_loop.md). - ./failed_migration.md sent you here. - Operator decision: post-deploy smoke tests failed and rolling forward isn't viable in the available window.

This is the code rollback procedure. It swaps the /opt/maid/current symlink back to the previous release. It does NOT roll back the database automatically. See the migration matrix below to decide whether a DB restore is also needed.

Detection

  • Manual operator decision following a deploy.
  • maid_engine_error_rate step-change after a release.
  • Player report cluster within minutes of a deploy.

Blast radius

  • Players affected: ALL on this instance, for the duration of the engine restart (~30-60 s assuming no migration involvement).
  • Data at risk: nothing if the new release was running cleanly with the prior schema (additive migrations only). For other migration classes, see the matrix.
  • AI/external systems: AI dialogue restarts; in-flight conversations lose context (Redis cache survives; PG-side conversation logs intact).

Prerequisites

  • Tools: maid-admin, systemctl, journalctl, ls, readlink.
  • Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units, which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) does not grant (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root).
  • Env file: /etc/maid/<inst>.env.
  • Source checkout (MAID_SRC): scripts/install.sh is repo-only and is not installed at /opt/maid/current/bin/ (staging is deferred to M1.4). export MAID_SRC=/path/to/maid-checkout before the rollback step; it runs sudo bash "${MAID_SRC}/scripts/install.sh" --rollback-version --yes.
  • Paths to know:
  • /opt/maid/current — symlink to active release
  • /opt/maid/versions/<version>/ — versioned release dirs
  • /opt/maid/versions/PREVIOUS — marker file naming the prior release (read by install.sh --rollback-version)
  • Audit access: tail -f /var/log/maid/ops-audit.jsonl.
  • Escalation: see ./escalation-contacts.md.template.

First 5 minutes (LITERAL commands)

# 1. Confirm what's deployed vs what we'd roll back to
ls -la /opt/maid/current /opt/maid/versions
readlink -f /opt/maid/current
cat /opt/maid/versions/PREVIOUS   # prior release name (marker file)

# 2. Notify connected players (functional) so the brief gap is friendly.
#    `maid ops maintenance on` is an M9 stub — it does NOT gate logins;
#    see ../deployment/player_comms.md.
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"rolling back, ETA 5m"}'

# 3. Capture state
sudo journalctl -u maid-engine@<inst>.service -n 200 \
  > ./incident-pre-rollback-$(date -u +%Y%m%dT%H%M%SZ).log
maid-admin status --instance <inst> --json \
  > ./incident-status-pre-rollback.json 2>&1 || true

# 4. Perform the rollback
#
# CRITICAL: use `--rollback-version`, NOT `--rollback`.
# `install.sh --rollback` is the per-instance UNINSTALL flag (destructive;
# wipes instance config, units, optionally the DB). The two flags are
# mutually exclusive (scripts/install.sh:1532). The CODE-rollback flag
# is `--rollback-version` — a BOOLEAN flag (scripts/install.sh:201) that
# reads the prior release name from /opt/maid/versions/PREVIOUS
# (scripts/install.sh:1330-1474, in particular 1341-1363) and atomically
# swaps the /opt/maid/current symlink to that release. It requires
# `--yes` for the live swap (CRITICAL #1 gate at scripts/install.sh:
# 1363-1370); `--dry-run` is harmless and bypasses the gate.
#
# Source of truth for the target release: /opt/maid/versions/PREVIOUS
# (maintained by M1.4 release tooling). Read it here only for human
# visibility — do NOT pass it as a positional arg; the script ignores
# positional args after `--rollback-version`.
PREV_VERSION="$(cat /opt/maid/versions/PREVIOUS 2>/dev/null || echo '<missing>')"
echo "About to roll back /opt/maid/current -> ${PREV_VERSION}"
sudo bash "${MAID_SRC}/scripts/install.sh" --rollback-version --yes
# install.sh --rollback-version --yes:
#   - reads /opt/maid/versions/PREVIOUS for the target release name
#   - validates that /opt/maid/versions/<release>/ exists and is usable
#   - swaps /opt/maid/current symlink to that release atomically
#   - audits the swap (see _audit_rollback_version in install.sh)
#   - leaves the engine running (does NOT restart it)
# Use --dry-run instead of --yes to preview without changing the symlink.
#
# > **DO NOT** run `install.sh --rollback` here. That is the per-instance
# > UNINSTALL flag and will wipe the instance's runtime state. See
# > scripts/install.sh:1532 for the mutex guard.

# 5. Verify symlinks swapped
ls -la /opt/maid/current
cat /opt/maid/versions/PREVIOUS   # the new "previous" (was current pre-swap)
readlink -f /opt/maid/current     # should now be the prior version

# 6. Restart the engine to pick up the rolled-back code
sudo systemctl restart maid-engine@<inst>.service

# 7. Watch startup
sudo journalctl -u maid-engine@<inst>.service -f
# Wait for "engine ready" + leader-lease acquired.

Investigation

  • If install.sh --rollback-version complains that the PREVIOUS marker is missing or empty (scripts/install.sh:1344-1356), the box has never deployed a second release (M1.4 release tooling owns the marker), or someone pruned it. You cannot rollback by symlink. Options:
  • Pull a previous tarball from the release archive (see RB18 step 3).

  • Or fix-forward.

  • If --rollback-version complains that the target release directory is missing (scripts/install.sh:1357-1361), the marker points to a release that has been deleted from /opt/maid/versions/. Same options as above.

  • If you invoke --rollback-version without --yes (and without --dry-run), install.sh refuses with "refusing to proceed without --yes" (scripts/install.sh:1363-1370). This is intentional — the swap restarts every active maid-engine@*.service.

  • If the engine fails to start on the rolled-back release, the previous release may itself have been broken (rare), or the DB schema is now ahead of the rolled-back engine. See the migration matrix.

Mitigation — Migration matrix

When a release includes a schema migration, rolling back the code does NOT automatically reverse the schema. Use this matrix to decide if a DB action is also needed:

Migration class in the rolled-back release DB action required
None (code-only change) none
Additive (add column / table / index) none — prior engine ignores the new objects
Rename phase-1 (new col + dual-write) none — prior engine still reads/writes old col
Rename phase-2 (drop old col after backfill) restore from backup — prior engine still expects old col
Type change (column type alter) restore from backup — prior engine expects old type
Destructive (drop col/table, mass UPDATE) restore from backup

If a restore is required, complete the symlink rollback above first (so the priv-helper and engine binaries match the older release), then follow ./restore.md.

Recovery

  • readlink -f /opt/maid/current points at the older release.
  • sudo systemctl status maid-engine@<inst>.service → active (running) with uptime growing.
  • maid-admin doctor --phase runtime --instance <inst> all green.
  • maid-admin status --instance <inst> --json | jq .build shows the rolled-back version.
  • /readyz returns 200.
  • Post the all-clear to players via the admin broadcast (maid ops maintenance off is an M9 stub — no login gate to lift).
  • Players notified per ../deployment/player_comms.md.

Post-incident

  • File ticket with: rolled-back version, target version, reason, total downtime, whether restore was also needed.
  • File a release-process follow-up: what should the pre-deploy smoke tests have caught? Add a regression test.
  • Confirm the failed release is NOT auto-redeployed (pin in the package manager / deploy pipeline) until the bug is fixed.

Escalation