Skip to content

Runbook: Failed migration on engine start

Prerequisites

  • Tools: maid-admin, the maid CLI (see the MAID_SRC note below — it is not on root/operator PATH; run it as "${MAID_SRC}/.venv/bin/maid"), systemctl, journalctl, psql, jq.
  • Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units and sudo -u maid-engine … service-user shells, none of which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) grants (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root). Plus maid_app_<inst> PG peer auth.
  • Source checkout (MAID_SRC): the maid CLI and deploy/scripts/backup.sh are repo-only and are not staged under /opt/maid/current (or on any PATH) until M1.4 — install.sh creates /opt/maid/venv empty (uv venv --no-project), so there is no system-wide maid binary. Before running the steps below, export MAID_SRC=/path/to/maid-checkout and build its venv once with sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv is at /opt/maid/current/.uv/bin/uv on installer hosts). The steps then invoke "${MAID_SRC}/.venv/bin/maid" and "${MAID_SRC}/deploy/scripts/backup.sh" as the maid-engine service user, forwarding MAID_SRC through sudo with --preserve-env=MAID_SRC (env_reset drops it otherwise). The same checkout also provides scripts/install.sh for the code-rollback step below (sudo bash "${MAID_SRC}/scripts/install.sh" --rollback-version --yes).
  • Env file: /etc/maid/<inst>.env.
  • Paths to know:
  • Migration runner state: PG tables _migration_history, _migration_rollback_log, _migration_checkpoints (packages/maid-engine/src/maid_engine/migrations/history.py)
  • Release directory: /opt/maid/current/
  • Previous release (for rollback): /opt/maid/previous/
  • Audit access: tail -f /var/log/maid/ops-audit.jsonl.
  • Escalation: see ./escalation-contacts.md.template.

Symptoms

Summary

  • sudo systemctl start maid-engine@<inst>.service exits non-zero; systemctl status shows failed (Result: exit-code).
  • Engine journal: 
    MigrationFailure: <migration_id> failed at <step>; head_at_failure=<id>
    followed by the engine refusing to enter running.
  • maid-admin status --instance <inst> reports engine_state=down.
  • /readyz red.

Detection

  • systemd Restart=on-failure exhausted (engine in failed state).
  • maid db status shows a pending or failed migration at head.
  • No built-in engine webhook fires a named engine_start_failed_migration event. The observability webhook bridge (MAID_OBSERVABILITY_WEBHOOK_*) only POSTs periodic metric snapshots; wire an external Prometheus/Alertmanager rule on the engine failed unit state (or maid db status) to page, and notify players manually with maid ops announce.

Blast radius

  • Players affected: ALL on this instance. Engine cannot start.
  • Data at risk: depends on the migration. Additive migrations that failed mid-way are usually safe to retry; rename phase-2 and type-change migrations may have left the DB in a partially migrated state.
  • AI/external systems: AI dialogue down with the engine.

Diagnostic Steps

First 5 minutes (LITERAL commands)

# 1. Notify players externally — the engine is already down, so the in-game
#    broadcast is unavailable and `maid ops maintenance` is an M9 stub. Post
#    to the configured webhook(s). `maid ops announce` reads
#    MAID_BRIDGES_WEBHOOK_URLS from its own process env, so source the
#    instance env with auto-export (a bare --preserve-env of a never-exported
#    var forwards nothing):
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a;
   exec "${MAID_SRC}/.venv/bin/maid" ops announce --severity warn \
     --message "failed migration, investigating"'

# 2. Capture the failure
sudo journalctl -u maid-engine@<inst>.service -n 500 \
  > ./incident-engine-$(date -u +%Y%m%dT%H%M%SZ).log

# 3. What does the migration table say? The `maid db` CLI has no --instance
#    flag; it reads MAID_DB_* from the instance env file, so load that first
#    (the same file the systemd unit loads via EnvironmentFile=).
sudo -u maid-engine --preserve-env=MAID_SRC bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/.venv/bin/maid" db status'
# Expect: head_at_failure=<id>, last_applied=<id>, pending=[<ids>]

# 4. What's the most recent applied migration class? (--format json, not --json)
sudo -u maid-engine --preserve-env=MAID_SRC bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/.venv/bin/maid" db status --format json' \
  | jq '.last_applied, .head_at_failure, .pending'

Investigation

Step 1: identify the failing migration's class

Look at the migration file (in the release tarball): 

ls /opt/maid/current/packages/*/migrations/ \
  | grep -E '<head_at_failure>' || true
Migration classes (per plan R10.A.x migration matrix):

Class Example Rollback strategy
additive add column, add table, add index retry forward; no DB action needed
rename phase-1 add new column, dual-write old+new retry forward; no DB action needed
rename phase-2 drop old column after backfill DB state is now partially changed; restore from backup
type change column type alters (int -> bigint, text -> jsonb) restore from backup
destructive drop column / table, mass UPDATE restore from backup

The migration filename / header should tag its class. If unclear, treat as destructive.

Step 2: was the failure transient?

Most additive migrations fail for transient reasons (lock_timeout, statement_timeout, deadlock with a stray session). Check: 

grep -E 'timeout|deadlock|conflicting' ./incident-engine-*.log
If yes, forward-retry is appropriate (see Mitigation: forward-fix).

Step 3: is the migration partially applied?

The migration framework tracks state in three internal tables: _migration_history, _migration_rollback_log, _migration_checkpoints (see packages/maid-engine/src/maid_engine/migrations/history.py). 

sudo -u maid_app_<inst> psql -d maid_<inst> -c "
  select namespace, sequence, name, status, applied_at, checksum
  from _migration_history
  order by applied_at desc nulls last
  limit 10;"

# Any rollback attempts (latest first)?
sudo -u maid_app_<inst> psql -d maid_<inst> -c "
  select namespace, sequence, status, executed_at, error
  from _migration_rollback_log
  order by executed_at desc nulls last
  limit 10;"

# Any orphaned checkpoints from a crashed long-running migration?
sudo -u maid_app_<inst> psql -d maid_<inst> -c "
  select namespace, sequence, checkpoint_name, created_at
  from _migration_checkpoints
  order by created_at desc
  limit 10;"
- status=pending with old applied_at/created_at → orphaned record from a crashed run. Clear before retry (see Mitigation). - status=failed → expected; rerun semantics depend on class. - status=applied → migration thinks it succeeded but engine still refused to start; the failure was downstream (e.g., schema-shape check). Investigate the engine log further.

Resolution Steps

Mitigation

Decision: - Additive / rename phase-1 + transient failure → forward-fix. - Rename phase-2 / type-change / destructive → rollback (this is the common case; the typical path).

Forward-fix (rare — only for additive + transient)

The maid db repair command exposes three corrective flags (see packages/maid-engine/src/maid_engine/cli/commands/db_migrate.py:596+): 

# NOTE: `maid db` has no --instance flag — it reads MAID_DB_* from
# /etc/maid/<inst>.env, loaded inline below (the same file the systemd unit
# loads via EnvironmentFile=).
# Clear any stuck 'pending' record from a prior crash
sudo -u maid-engine --preserve-env=MAID_SRC bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/.venv/bin/maid" db repair --clear-pending'

# (Optional) Drop orphaned checkpoints from a partially-resumed run
sudo -u maid-engine --preserve-env=MAID_SRC bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/.venv/bin/maid" db repair --cleanup-checkpoints'

# (Optional) Re-compute file checksums if a CI patch changed migration
# bytes after the row was recorded
sudo -u maid-engine --preserve-env=MAID_SRC bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/.venv/bin/maid" db repair --fix-checksums'

# Retry
sudo -u maid-engine --preserve-env=MAID_SRC bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/.venv/bin/maid" db migrate'
# Expect: "applied <id>" or "no pending migrations"

# Start the engine
sudo systemctl start maid-engine@<inst>.service
maid-admin doctor --phase runtime --instance <inst>

If retry also fails, go to rollback.

Rollback (typical path)

This rolls the code back to the previous release. If the failed migration was destructive / type-change / rename phase-2, you also need to restore the DB.

# 1. Roll back the code (BOOLEAN flag; reads /opt/maid/versions/PREVIOUS
#    per scripts/install.sh:1341-1370; requires --yes for live swap)
sudo bash "${MAID_SRC}/scripts/install.sh" --rollback-version --yes
# See ./rollback.md for verification. Do NOT use bare `--rollback` — that
# is the per-instance UNINSTALL flag (destructive).

# 2. If migration was rename phase-2 / type-change / destructive: restore DB
#    (the migration matrix in ./rollback.md tells you when this is needed)
# Engine is down; notify players externally via webhook (`maid ops
# maintenance` is an M9 stub). Source the instance env with auto-export so
# announce inherits MAID_BRIDGES_WEBHOOK_URLS (a bare --preserve-env of a
# never-exported var forwards nothing):
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a;
   exec "${MAID_SRC}/.venv/bin/maid" ops announce --severity warn \
     --message "rolling back failed migration"'
# Pick the latest verified backup taken BEFORE the failed migration ran.
# `backup.sh list --json` prints an array of backup-id strings (each id is
# timestamp-prefixed, so lexical order is chronological). Pick the newest id
# taken BEFORE the failed migration ran, then verify it with `verify --backup-id`.
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance <inst> --json' \
  | jq -r '.[]' | sort
# Then follow ./restore.md.

# 3. Start the engine (now on the prior release)
sudo systemctl start maid-engine@<inst>.service
maid-admin doctor --phase runtime --instance <inst>
# All-clear to players (engine is back up). `maid ops maintenance off` is an
# M9 stub; use the functional in-game broadcast:
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"migration resolved; service restored"}'

Recovery

  • maid-admin doctor --phase runtime --instance <inst> all green.
  • maid-admin status --instance <inst> engine_state=running.
  • maid db status (run with /etc/maid/<inst>.env sourced) shows no failed/pending migrations relative to the currently-running release.
  • /readyz returns 200.
  • Players notified per ../deployment/player_comms.md.

Post-incident

  • File ticket with: failed migration id, class, root cause, forward vs rollback chosen, total downtime.
  • If forward-fix path used, file a follow-up to add the missing guard (lock_timeout, dependency check) that would have prevented the transient failure.
  • If rollback path used, the failed migration needs a redesign before being retried in the next release — it cannot just be re-run as-is.
  • Update this runbook if the migration table query above did not give you what you needed.

Escalation