Skip to content

Runbook: Restore from backup (from-scratch)

Prerequisites

  • Interim maid CLI (MAID_SRC): there is no system-wide maid binary until M1.4 (install.sh leaves /opt/maid/venv empty), so the CLI runs from a source checkout. export MAID_SRC=/path/to/maid-checkout and build its venv once: sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv: /opt/maid/current/.uv/bin/uv). Commands below run "${MAID_SRC}/.venv/bin/maid" as the maid-engine user; inside bash -c blocks MAID_SRC is forwarded via --preserve-env=MAID_SRC.
  • Tools: maid-admin, systemctl, journalctl, psql, jq, enough disk on /var/lib/maid-restore/<inst>/ for a full staging cluster (≥ current DB size + 20 % headroom).
  • Access: root, or an equivalently broad sudo grant — the functional restore path runs sudo systemctl on the instance units, sudo -u maid-engine …, and sudo -u postgres … shells, none of which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) grants (it only covers fixed maid-admin <verb> calls, and maid-admin restore is an M1.3 shim that does not restore). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root).
  • Env file: /etc/maid/<inst>.env.
  • Source checkout: deploy/scripts/backup.sh and deploy/scripts/restore.sh are repo-only. The M1.4 release tarball populates /opt/maid/current/{bin,lib,packages} but not deploy/scripts/, so /opt/maid/current/deploy/scripts/… does not exist on an installed host. Run these drivers from a MAID source checkout matching the deployed release tag and point MAID_SRC at that checkout root: export MAID_SRC=/path/to/maid-checkout. (restore.sh also sources scripts/lib/distro.sh, so use a full checkout — not just deploy/.)
  • Paths to know:
  • /var/lib/maid-backups/<BID>/ — local staging copy of a backup, if present. backup.sh stages under ${MAID_BACKUP_LOCAL_DIR}/<BID> (default /var/lib/maid-backups/<BID>, no per-instance subdir; the instance is embedded in the <BID> itself) and prunes it after upload, so a completed backup usually lives only on the transport.
  • /var/lib/maid-restore/<inst>/ — staging cluster work dir
  • /var/lib/postgresql/<ver>/main/ or /var/lib/pgsql/<ver>/data/ — production cluster
  • Audit access: tail -f /var/log/maid/ops-audit.jsonl.
  • Escalation: see ./escalation-contacts.md.template.

Symptoms

Summary

You are reading this because one of: - ./db_down.md sent you here after a PANIC / corruption. - ./failed_migration.md rollback path requires a rename phase-2 or type-change reversal. - ./RB18_host_loss_dr.md called this in step "Restore Postgres" of a host-loss DR. - An operator manually triggered a restore drill (good — do these!).

This is the destructive path. The current PG data directory of the target instance WILL be replaced.

Read ./backup_failed.md first if you are not sure the chosen backup is verified. Restoring from a corrupt backup is worse than the original incident.

Detection

  • Manual decision based on another runbook's referral.
  • The candidate backup passes backup.sh verify — i.e. the driver exits 0 (every file listed in MANIFEST.json re-downloaded and sha256-matched), not merely that some line printed. Exit 8 means a listed file is missing/corrupt; do not restore from it. Check it env-sourced (backup.sh reads MAID_DEPLOY_BACKUP_REMOTE from its own process env, so a bare sudo aborts rc=2):
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance <inst> --backup-id <BID>'
# rc=0 → verified; rc=8 → corrupt/missing file, pick an earlier backup.

Blast radius

  • Players affected: ALL on this instance, for the full duration of the restore.
  • Data at risk: everything written since the chosen backup. The longer the restore, the more "your community lost X hours of progress" you have to explain. Pick the latest verified backup whose contents pre-date the incident.
  • AI/external systems: NPC memory and dialogue logs reset to the backup state; players will notice NPCs "forgetting" things from the lost window.

Diagnostic Steps

First 5 minutes (LITERAL commands)

# 1. Pick the backup. `list --json` prints a JSON array of backup-id
#    strings (timestamp-prefixed, so they sort chronologically).
#    backup.sh reads MAID_DEPLOY_BACKUP_REMOTE from its OWN process env,
#    so source the instance env with auto-export inside the maid-engine
#    shell (a bare sudo does not load EnvironmentFile=; source_transport
#    would abort with "MAID_DEPLOY_BACKUP_REMOTE is not set").
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance <inst> --json' \
  | jq -r '.[]' | sort -r

# Set the chosen BID
export BID=<backup-id>
export INST=<instance>

# 2. Warn players. maid ops maintenance is an M9 stub (no login gate);
#    during a restore the engine is stopped anyway. Announce to webhooks
#    (functional, webhook-only — export the URL into this shell first):
export MAID_BRIDGES_WEBHOOK_URLS="discord:https://discord.com/api/webhooks/<id>/<token>"
sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --channels webhook --severity warn \
  --message "Emergency restore in progress. Service down ~60m. See status page for updates."
# (If the engine is still up, also broadcast in-game:
#   curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
#     -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
#     -d '{"prefix":"[MAINTENANCE]","message":"Emergency restore in progress; service down ~60m."}')

# 3. Capture pre-restore state
sudo journalctl -u maid-engine@"$INST".service -n 200 \
  > ./incident-engine-pre-restore-$(date -u +%Y%m%dT%H%M%SZ).log
sudo systemctl status maid-engine@"$INST".service --no-pager \
  > ./incident-engine-status-pre-restore.txt

# 4. Stop the engine
sudo systemctl stop maid-engine@"$INST".service

Resolution Steps

Investigation (sanity checks before destruction)

# 5. Confirm the chosen backup is present locally (pull from remote if not).
#    Local staging is /var/lib/maid-backups/<BID> (no per-instance subdir;
#    the instance is embedded in <BID>) and is pruned after upload, so it
#    is often absent — that is fine, the restore.sh staging step below
#    re-fetches it from the configured transport automatically.
sudo ls -la /var/lib/maid-backups/"$BID"/ 2>/dev/null || \
  echo "not staged locally — restore.sh will fetch it from the transport"
sudo cat /var/lib/maid-backups/"$BID"/MANIFEST.json 2>/dev/null | head -30 || true

# If it is not present locally, don't worry: the `restore.sh staging`
# step below fetches the backup from the configured transport automatically.

# 6. Verify integrity. INST and BID were exported above; forward them
#    into the maid-engine shell and source that instance's env so
#    backup.sh sees MAID_DEPLOY_BACKUP_REMOTE.
sudo -u maid-engine --preserve-env=INST,BID,MAID_SRC bash -c \
  'set -a; . "/etc/maid/${INST}.env"; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance "${INST}" --backup-id "${BID}"'
# Gate on the EXIT CODE, not on any text — backup.sh verify prints a
# "verify summary: N PASS, N FAIL" line, NOT "verify OK". rc=0 means every
# file listed in MANIFEST.json was re-downloaded and sha256-matched; rc=8
# means a listed file is missing/corrupt (or the manifest itself failed its
# sidecar check). On any non-zero rc, STOP and pick an earlier backup:
#   if ...verify...; then echo "verified"; else echo "CORRUPT — pick older"; fi

Mitigation (the restore itself)

Step 1: stage the restore (non-destructive)

# NOTE: the `maid-admin restore --instance <inst> --backup-id <BID>`
# priv-helper verb is an M1.3 SHIM — it validates the args, takes the
# restore flock, writes an audit row, then returns EX_DEFERRED (2) with
# "restore engine ships in M1.4" (see packaging/admin/priv_helper.py
# verb_restore). It does NOT stage or restore a database yet.
#
# Until M1.4 wires the priv-helper path, stage the restore by invoking
# the functional restore.sh directly as root, from a source checkout
# (deploy/scripts/ is NOT installed under /opt/maid/current — set MAID_SRC
# per Prerequisites). restore.sh calls source_transport for ALL modes, so
# the instance env (MAID_DEPLOY_BACKUP_REMOTE) must be in its process env
# or it aborts rc=2 — source it with auto-export. INST/BID were exported
# above; forward INST/BID/MAID_SRC into the root shell.
sudo --preserve-env=INST,BID,MAID_SRC bash -c \
  'set -a; . "/etc/maid/${INST}.env"; set +a; exec "${MAID_SRC}/deploy/scripts/restore.sh" staging --instance "${INST}" --backup-id "${BID}"'
# restore.sh staging:
#   - fetches + verifies the backup from the transport, extracts the base
#     into a fresh cluster at <PG_DATA_BASE>/data-restore-<inst>
#   - starts that cluster on an alternate port, read-only smoke-test
#   - does NOT touch the production cluster
Watch (restore.sh logs to the journal under the maid-restore tag, not a maid-restore@ unit): 
sudo journalctl -t maid-restore -f
# Wait for "PASS  staging cluster ready on port <N>" and an
# OPS_RESTORE_STAGING_COMPLETE audit row in /var/log/maid/ops-audit.jsonl.

The staging cluster name defaults to restore-<inst> (unless you passed --target-cluster NAME); promote needs it as a non-empty token (promote derives the data dir from the instance, so the value only has to match what staging reported): 

export STAGING=restore-<inst>

Step 2: verify the staging cluster

# Doctor against the staging cluster
# NOTE: `--staging-cluster` is not exposed by the M1.3 maid-admin wrapper
# (see packaging/admin/wrapper.py:332-396) — this invocation is a TODO
# placeholder for M1.4. Today, point doctor at the staging port via the
# instance's env file pinned to the staging cluster, or skip this step
# and rely on the engine-table spot-check below.
# TODO(M1.4): wire --staging-cluster (or equivalent) into maid-admin doctor.
maid-admin doctor --phase runtime --instance "$INST"

# Spot-check engine-level tables (the only schema present in core
# maid-engine; see packages/maid-engine/src/maid_engine/migrations/engine/).
# Content-pack-specific tables (e.g. classic-rpg's entities/world_rooms)
# are layered on top by pack migrations and may not exist or may have
# different names depending on which packs the restored instance loads.
sudo -u postgres psql -h /var/run/postgresql -p <staging-port> -d maid_"$INST" -c "
  -- accounts: 0001_initial_schema.py
  select count(*) as accounts from accounts;
  select max(created_at) as newest_account from accounts;
  -- sessions: 0001_initial_schema.py
  select count(*) as sessions from sessions;
  select max(created_at) as newest_session from sessions;
  -- documents: 0002_document_store.py
  select count(*) as documents from documents;
  select max(updated_at) as latest_write from documents;
"
# 'latest_write' tells you the recovery point against the engine's
# generic document store.

# Optional: validate any installed content packs against their declared
# schemas if `maid pack validate` is wired in this build. (As of M1.3
# this is not implemented; treat as TODO.)
# TODO(M1.3+): `maid pack validate <name> --instance "$INST" --staging-cluster ...`

Stop and think: is the recovery point earlier than the incident trigger? If not, the incident may have already corrupted this backup. Pick an earlier one.

Step 3: promote the staging cluster to production

# restore.sh calls source_transport for ALL modes (promote included),
# so MAID_DEPLOY_BACKUP_REMOTE must be in its process env or it aborts
# rc=2. A bare sudo does not load EnvironmentFile= — source the instance
# env with auto-export and forward the shell vars (INST/STAGING were
# exported earlier; MAID_SRC per Prerequisites):
sudo --preserve-env=INST,STAGING,MAID_SRC bash -c \
  'set -a; . "/etc/maid/${INST}.env"; set +a; exec "${MAID_SRC}/deploy/scripts/restore.sh" promote --instance "${INST}" --staging-cluster "${STAGING}"'
# restore.sh promote (running as root):
#   - re-checks the leader lease and refuses to swap if a live engine
#     re-appeared
#   - stops maid-engine@<inst>, MASKS it (systemctl mask) AND drops in
#     /etc/systemd/system/maid-engine@<inst>.service.d/95-restore-mask.conf
#     (ExecStartPre=/bin/false) so it cannot auto-start against the
#     half-swapped cluster — you MUST clear BOTH in Step 4
#   - stops postgresql.service, moves the current data dir aside as
#     <data>.pre-restore.<ts>/, moves the staged data dir
#     (<PG_DATA_BASE>/data-restore-<inst>) into the production location
#   - starts postgresql.service and runs a best-effort doctor prestart
#   - leaves maid-engine@<inst> masked (Step 4 unmasks it)

Step 4: unmask and start the engine

# Step 3's `restore.sh promote` masked maid-engine@<inst> TWO ways, and
# BOTH must be cleared before it will start:
#   (a) `systemctl mask` symlink (→ /dev/null) — remove with systemctl unmask
#   (b) 95-restore-mask.conf drop-in (ExecStartPre=/bin/false) — remove with
#       `maid-admin restore --unmask` (priv_helper _restore_unmask; it only
#       deletes the drop-in + daemon-reload, it does NOT run systemctl unmask,
#       so you need both commands).
sudo systemctl unmask maid-engine@"$INST".service

# The wrapper marks --backup-id required even for --unmask (the value is
# ignored on the unmask path); pass any placeholder.
maid-admin restore --unmask --instance "$INST" --backup-id 00000000
# Removes /etc/systemd/system/maid-engine@<inst>.service.d/95-restore-mask.conf
# and reloads systemd (idempotent — safe to re-run).

sudo systemctl daemon-reload
sudo systemctl start maid-engine@"$INST".service

# Watch for clean startup
sudo journalctl -u maid-engine@"$INST".service -f
# Wait for "engine ready" / leader-lease acquired.

Step 5: post-promotion verification

maid-admin doctor --phase runtime --instance "$INST"
# All green.

maid-admin status --instance "$INST" --json | jq '.engine_state, .db, .persistence'
# engine_state=running, db=reachable, persistence.queue_depth=0

# Confirm leader-lease (schema per
# packaging/postgres/migrations/0001_leader_lock.sql)
sudo -u maid_app_"$INST" psql -d maid_"$INST" -c "
  select holder_pid, generation,
         leased_at, lease_expires_at,
         lease_expires_at - now() as ttl_remaining
  from leader_lock;"
# ttl_remaining MUST be positive; cross-check that the holder PID is
# this host's engine:
sudo -u maid_app_"$INST" psql -d maid_"$INST" -c "
  select sa.pid, sa.application_name, sa.client_addr
  from pg_stat_activity sa
  join leader_lock ll on ll.holder_pid = sa.pid
  where sa.application_name LIKE 'maid-engine:%';"

Step 6: end maintenance

# maid ops maintenance off is an M9 stub — nothing to toggle; starting
# the engine restored normal logins. Announce the all-clear to webhooks.
# `maid ops announce` reads MAID_BRIDGES_WEBHOOK_URLS from its own process
# env; source the instance env with auto-export (a bare --preserve-env of a
# never-exported var forwards nothing):
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a;
   exec "${MAID_SRC}/.venv/bin/maid" ops announce --severity info \
     --message "Restore complete. Service restored. Apologies for the disruption."'
# Also tell connected players in-game:
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"Restore complete — service restored."}'

"If it goes wrong" branches

Staging fails (STAGING FAILED)

  • Look at journalctl -t maid-restore -n 200 (restore.sh logs under the maid-restore syslog tag; there is no maid-restore@ systemd unit).
  • Common causes:
  • Disk full on /var/lib/maid-restore/ → free space, retry.
  • Backup file corrupt → pick an earlier verified backup, retry from Step 1.
  • PG version mismatch (backup from PG N, host has PG N-1) → install matching PG version per RB18 then retry.
  • Production cluster is untouched; you can safely retry.

Promotion fails mid-swap

  • Production data dir was moved to <data>.pre-restore.<ts>/ (dot before the <ts> timestamp, e.g. main.pre-restore.20250115T031500Z).
  • restore.sh promote logs the exact recovery mv on failure. If it didn't: 
    sudo systemctl stop postgresql || true
sudo mv /var/lib/postgresql/<ver>/main /var/lib/postgresql/<ver>/main.failed-restore.$(date -u +%s)
sudo mv /var/lib/postgresql/<ver>/main.pre-restore.<ts> /var/lib/postgresql/<ver>/main
sudo systemctl start postgresql
  • This reverts to the pre-restore state (still broken — you came here because the original DB was bad — but at least no worse). Escalate.

Engine refuses to start after promote

  • Check the engine log; it may be a migration mismatch (the backup was taken at schema version N; engine at version N+2). See ./failed_migration.md. Usually the fix is to deploy the matching engine release; the backup's MANIFEST.json records the engine version (code_version) that produced it.

Players report data weirdness post-restore

  • Expected: NPCs forget recent events; quest progress reverts; auction house listings rewind. Communicate clearly.
  • If something is wrong beyond expected backup-time rollback, capture details before any further restore. Do NOT layer a second restore on top of the first.

Recovery

  • maid-admin doctor --phase runtime --instance <inst> all green.
  • /readyz returns 200.
  • maid-admin status --instance <inst> engine_state=running.
  • Leader-lease heartbeat refreshing within LEASE_TTL_S (30 s).
  • Players able to connect and interact.

Post-incident

  • File ticket with: BID restored, recovery point (latest write timestamp in the backup), total downtime, what triggered the restore, any deviation from this runbook.
  • Schedule a fresh backup right away: 
    # NOTE: on a stock host this fails 203/EXEC until M1.4 stages
# bin/backup.sh; until then run ./deploy/scripts/backup.sh full from a
# checkout instead (see backup.md "Manual backup (interim)").
sudo systemctl start maid-backup@<inst>.service
  • Update ./escalation-contacts.md.template "Last drilled" date if this was a drill.
  • Update this runbook with anything that didn't go as written.

Escalation