Skip to content

Runbook: RB18 — Host-loss disaster recovery

Prerequisites

  • Interim maid CLI (MAID_SRC): there is no system-wide maid binary until M1.4 (install.sh leaves /opt/maid/venv empty), so the CLI runs from a source checkout. export MAID_SRC=/path/to/maid-checkout and build its venv once: sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv: /opt/maid/current/.uv/bin/uv). Commands below run "${MAID_SRC}/.venv/bin/maid" as the maid-engine user; inside bash -c blocks MAID_SRC is forwarded via --preserve-env=MAID_SRC.
  • Tools (on the NEW box, after provisioning): wget, tar, sudo, systemd, psql, ssh, sha256sum, python3.12, uv (see ./RB22_python_uv.md if those are not present).
  • Tools (on the side that holds the off-host backup): ssh, sha256sum.
  • Access:
  • Hosting console for the new box (URL in ./escalation-contacts.md.template).
  • SSH key + path to the off-host backup destination (URL in ./escalation-contacts.md.template).
  • DNS registrar (to flip the player-facing hostname; URL in ./escalation-contacts.md.template).
  • Env file: you'll re-create /etc/maid/<inst>.env from your secrets store; have the path documented.
  • Audit access: tail -f /var/log/maid/ops-audit.jsonl (will not exist on the new box until install.sh runs).
  • Escalation: see ./escalation-contacts.md.template.

Symptoms

Summary

The MAID host is gone or unrecoverable. Specifically: - Hosting provider reports physical failure (disk array dead, board fault, network port dead). - Host is unreachable via SSH and console; provider cannot bring it back in a useful timeframe. - Filesystem corruption beyond fsck recovery; local backups also inaccessible. - Compromise suspected; you want to rebuild on a clean host rather than try to clean the existing one.

You are rebuilding MAID from off-host artifacts on a NEW box.

Target RTO: ≤ 30 minutes from "new box has SSH and uv" to "engine serving players on the rolled-back data". The wall-clock total is longer because provisioning the box itself is provider- dependent.

Detection

  • Operator decision after escalation.
  • Hosting provider notification.
  • ./two_instances_detected.md cross-host branch that determined the original host is dead.

Blast radius

  • Players affected: ALL on this instance, for the full DR window.
  • Data at risk: everything written since the last off-host backup that survived the loss. That backup IS your recovery point. Pick the latest one whose remote copy you can prove sha256-verified.
  • AI/external systems: AI dialogue is offline with the engine. NPC memory rewinds to the backup's recovery point.

Diagnostic Steps

Before committing to a full DR rebuild (≥30 min, players offline, backup-RPO data loss), prove the primary really is gone and that you have a usable recovery point. Five minutes here can save an hour of unneeded rebuild if the primary is just briefly unreachable.

Confirm the primary is truly dead

# 1. Network reachability (run from at least two vantage points if you
#    have them — your workstation AND a second cloud).
export OLDHOST=<fqdn-or-ip-of-dead-box>
ping -c 4 -W 2 "$OLDHOST" || echo "ICMP: dead"
ssh -o ConnectTimeout=5 -o BatchMode=yes "$OLDHOST" \
    'hostname; systemctl is-active maid-engine@*.service' \
  || echo "SSH: dead"

# 2. If the box answers SSH but the engine is dead, this is NOT a host
#    loss — use ./db_down.md or ./oom_loop.md instead.
# 3. If the hosting console shows the VM "running" but unreachable,
#    try a hard reboot from the console BEFORE declaring host loss.
#    DR from off-host backup is destructive to recent writes; avoid it
#    if a reboot recovers the box.

Confirm DNS state

# Your players' DNS should still point at the dead host. Verify before
# DR so you know exactly what records need updating after promotion.
dig +short maid.<your-domain>.tld
dig +short -x "$OLDHOST"   # reverse — sanity check
# Record the current TTL so you can predict propagation delay later.
dig +noall +answer maid.<your-domain>.tld | awk '{print $2}'

Locate the recovery point (latest verified off-host backup)

# 4. Off-host backup reachability from your workstation. The transport
#    writes every backup as <remote-root>/<BID> (no per-instance subdir;
#    the instance is embedded in <BID>), so list the root directly.
ssh -i ~/.ssh/maid_backup_id_ed25519 -o BatchMode=yes -o ConnectTimeout=5 \
    <backup-user>@<backup-host> 'ls -la /srv/maid-backups/ | head'

# 5. List backups present on the transport and pick the newest. On a DR
#    (primary host gone) the backup host is just the ssh:// storage
#    destination and may not have MAID installed, so list the backup
#    directories directly instead of relying on backup.sh being present
#    there. Each backup is a directory named by its id
#    (<timestamp>-<instance>-<rand>, so names sort chronologically and
#    carry the instance). Backups for all instances share the root, so
#    filter by the instance embedded in the id.
export INST=<instance>
ssh -i ~/.ssh/maid_backup_id_ed25519 <backup-user>@<backup-host> \
    "ls -1 /srv/maid-backups/" \
  | grep -- "-${INST}-" | sort -r | head -1
# Capture the chosen backup_id for the next phase.
export BID=<backup-id-from-above>
echo "recovery point: BID=$BID INST=$INST"

If list returns no backup ids, escalate immediately — DR is risky and may complete with corrupted state. The chosen $BID is sha256-verified explicitly in a later step. See ./backup_failed.md for why a backup might not be verified, and ./escalation-contacts.md.template for who to call before proceeding.

Resolution Steps

First 5 minutes (LITERAL commands — on YOUR workstation)

# 1. Declare extended outage to players (DNS still points at dead box
#    so use Discord / status page / email)
# Status-page snippet template lives in ../deployment/player_comms.md.
echo "Post in Discord / status page: host failure, ETA 30-60m once new host is up."

# 2. Provision a new box at your hosting provider per the access guide.
#    Capture its IP/hostname.
export NEWHOST=<ip-or-hostname-of-new-box>

# 3. Re-export the diagnostics you captured above so the rest of this
#    procedure can use them ($INST, $BID — set during Diagnostic Steps).
echo "will restore BID=$BID for INST=$INST onto $NEWHOST"

DR procedure (on the NEW box, via SSH)

Step 1: bootstrap MAID — install but don't start

# SSH in (this crosses into the NEW box — shell variables you exported on
# your workstation do NOT survive the hop, so re-establish them here).
ssh root@$NEWHOST          # or your provisioned admin user

# Re-establish the identifiers you recorded during Diagnostic Steps.
# Type the literal values in — nothing carried across the SSH boundary.
export INST=<instance>
export BID=<backup-id-you-chose>

# Get the MAID release tarball corresponding to the backup's engine version
# (the backup MANIFEST.json records its engine version in `code_version`; match it).
wget https://<your-release-host>/maid/<version>/maid-<version>.tar.gz
tar -xzf maid-<version>.tar.gz
cd maid-<version>/

# Point MAID_SRC at this extracted release tree. The backup/restore drivers
# live under deploy/scripts/ here (they are NOT installed under
# /opt/maid/current — the release tarball only populates bin/lib/packages),
# and restore.sh also sources scripts/lib/distro.sh, so use this full tree.
export MAID_SRC="$PWD"

# Install. The installer is scripts/install.sh and REQUIRES an instance
# (positional <instance> or --instance NAME); omitting it aborts EX_USAGE.
# --no-start makes it provision OS, users, PG, systemd units, etc. and stop
# after the PG stage without starting the engine — important here, because
# starting on an empty DB would acquire the lease (a problem if the original
# lease entry is still alive on the dead host's last DB state). See
# scripts/install.sh (usage) and the --no-start gate (scripts/install.sh:202).
sudo ./scripts/install.sh --instance "$INST" --no-start

# Confirm the engine unit is installed but inactive
sudo systemctl is-enabled maid-engine@$INST.service   # should be: enabled
sudo systemctl is-active  maid-engine@$INST.service   # should be: inactive

Step 2: stop anything that might race the restore

sudo systemctl stop 'maid-engine@*'
sudo systemctl stop 'maid-backup@*'

Step 3: restore /etc/maid env from your secrets store

# Restore /etc/maid/<inst>.env from your secrets store (1Password / SOPS / etc.)
# This must contain MAID_DB_*, MAID_AI_*, MAID_BACKUP_*, etc.
# Ownership MUST be root:maid-engine mode 0640 — the same as install.sh
# writes (install.sh:829-832). The engine runs as maid-engine and the
# env-wrapped `sudo -u maid-engine … . $INST.env` calls below must be able
# to read it; group maid-admin would leave maid-engine unable to read it.
sudo install -m 0640 -o root -g maid-engine /path/to/secret/$INST.env /etc/maid/$INST.env
sudo ls -la /etc/maid/$INST.env   # expect: -rw-r----- root maid-engine

Step 3b: provision the backup transport SSH key

# The transport authenticates to the off-host backup destination with
# /var/lib/maid-engine/.ssh/id_ed25519 (transport-ssh.sh:63). install.sh
# does NOT generate this key, and a fresh key would not be authorized on
# the backup host — so restore the ORIGINAL private key from your secrets
# store. Its public half is already in the backup host's authorized_keys,
# so transport ops (verify / restore.sh / backup) work immediately.
sudo install -d -o maid-engine -g maid-engine -m 0700 /var/lib/maid-engine/.ssh
sudo install -m 0600 -o maid-engine -g maid-engine \
    /path/to/secret/backup-id_ed25519 /var/lib/maid-engine/.ssh/id_ed25519
# transport-ssh.sh enforces owner=maid-engine and mode 600/400 (dies rc=4
# otherwise). known_hosts is created on first connect via
# StrictHostKeyChecking=accept-new; just ensure the dir exists (above).
sudo -u maid-engine stat -c '%U %a %n' /var/lib/maid-engine/.ssh/id_ed25519
# expect: maid-engine 600 /var/lib/maid-engine/.ssh/id_ed25519

Step 4: verify the chosen backup over the transport

# You do NOT need to copy the backup down by hand: `backup.sh verify`
# re-downloads MANIFEST.json + every file from the transport into a temp
# dir and sha256-checks them (backup.sh mode_verify), and `restore.sh
# staging` (Step 5) re-fetches the base itself. Both use the transport key
# provisioned in Step 3b, as the maid-engine user.
#
# Verify integrity. INST and BID were exported in Step 1; forward them into
# the maid-engine shell and source this host's instance env so backup.sh
# sees MAID_DEPLOY_BACKUP_REMOTE (a bare sudo does not load it). MAID_SRC
# was set in Step 1 to the extracted release tree.
sudo -u maid-engine --preserve-env=INST,BID,MAID_SRC bash -c \
  'set -a; . "/etc/maid/${INST}.env"; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance "${INST}" --backup-id "${BID}"'
# Gate on the EXIT CODE, not on text — backup.sh verify prints a
# "verify summary: N PASS, N FAIL" line, NOT "verify OK". rc=0 → every file
# in MANIFEST.json re-downloaded and sha256-matched; rc=8 → a listed file is
# missing/corrupt. On any non-zero rc, pick an earlier BID and retry.

Step 5: stage + promote the restore

This is the same procedure as ./restore.md, but on a fresh PG cluster that has no production data to swap with. Because the maid-admin restore priv-helper verb is still an M1.3 shim, run the functional restore.sh directly (as below).

# Stage. The `maid-admin restore` verb is an M1.3 SHIM that defers to
# M1.4 (returns EX_DEFERRED); it does NOT stage a DB. Invoke the
# functional restore.sh directly as root instead. source_transport runs
# for ALL modes, so source the instance env with auto-export (INST/BID/
# MAID_SRC exported in Step 1).
sudo --preserve-env=INST,BID,MAID_SRC bash -c \
  'set -a; . "/etc/maid/${INST}.env"; set +a; exec "${MAID_SRC}/deploy/scripts/restore.sh" staging --instance "${INST}" --backup-id "${BID}"'
# Watch: sudo journalctl -t maid-restore -f  → wait for
# "PASS  staging cluster ready". The staging cluster name defaults to
# restore-<inst>.
export STAGING=restore-$INST

# Promote (restore.sh calls source_transport for ALL modes, so the
# instance env — MAID_DEPLOY_BACKUP_REMOTE — must be in its process env;
# a bare sudo does not load it. INST/STAGING/MAID_SRC were exported above.)
sudo --preserve-env=INST,STAGING,MAID_SRC bash -c \
  'set -a; . "/etc/maid/${INST}.env"; set +a; exec "${MAID_SRC}/deploy/scripts/restore.sh" promote --instance "${INST}" --staging-cluster "${STAGING}"'

# restore.sh promote masks maid-engine@<inst> TWO ways — a `systemctl mask`
# symlink AND a 95-restore-mask.conf drop-in (ExecStartPre=/bin/false) — so
# BOTH must be cleared before Step 7 will start the engine:
sudo systemctl unmask maid-engine@$INST.service
# --unmask removes the drop-in (priv_helper _restore_unmask; it does NOT
# run systemctl unmask, hence both commands). --backup-id is required by
# the wrapper even for --unmask (the value is ignored on that path).
maid-admin restore --unmask --instance $INST --backup-id 00000000
sudo systemctl daemon-reload

Step 6: clear the stale leader lease from the OLD host

The restored DB contains the dead host's leader_lock row. Let it expire naturally — the takeover predicate lease_expires_at < now() runs on the new engine's first start and atomically bumps generation. Do NOT manually delete the row. 

# Optional: inspect the inherited lease state (schema per
# packaging/postgres/migrations/0001_leader_lock.sql).
sudo -u maid_app_$INST psql -d maid_$INST -c "
  select holder_pid, generation,
         leased_at, lease_expires_at,
         lease_expires_at - now() as ttl_remaining
  from leader_lock;"

# Wait for the takeover window — `lease_expires_at` must be <= now().
# Since the backup was taken at least many minutes ago (`leased_at`
# reflects the dead host's last renewal), this is already satisfied.
# The engine's takeover predicate runs on start.
sleep 5

Step 7: start the engine

sudo systemctl start maid-engine@$INST.service

# Watch startup
sudo journalctl -u maid-engine@$INST.service -f
# Expect:
#   - takeover predicate fires, lease acquired with new generation
#   - migrations applied (none pending, hopefully)
#   - engine ready

Step 8: verify the new host

maid-admin doctor --phase runtime --instance $INST
# Every check must be pass.

maid-admin status --instance $INST --json | jq '
  {engine_state, build, db, persistence, ai, lease: .leader_lock}'

curl -fsS http://localhost:9090/readyz   # expect 200 (internal observability server)

Per-protocol verification per ./maintenance_window.md "Per-protocol verification" section.

Step 9: flip DNS to the new host

# Update A/AAAA record for play.<your-domain> at your DNS registrar
# (URL in escalation-contacts.md.template). TTL should already be low (<= 300s)
# per the deployment guide.
dig +short play.<your-domain>      # confirm new IP propagated

Step 10: re-enable backups on the new box

# Take a fresh backup so the new host has a current recovery point.
# The stock maid-backup@<inst> timer + service are NOT functional on an
# installed host: their ExecStart is @@MAID_HOME@@/bin/backup.sh
# (/opt/maid/current/bin/backup.sh), which the release tarball does NOT
# install, so both fail 203/EXEC. Do NOT rely on the timer here. Run the
# functional driver from the extracted release tree instead (MAID_SRC was
# set in Step 1); it uses the transport key from Step 3b as maid-engine.
sudo -u maid-engine --preserve-env=INST,MAID_SRC bash -c \
  'set -a; . "/etc/maid/${INST}.env"; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" full --instance "${INST}"'
# Wait for it to print SUCCESS (see backup.md "Manual backup (interim)").
# Verify the fresh backup we just created (newest id on the transport).
# Source this host's instance env inside the maid-engine shell so
# backup.sh sees MAID_DEPLOY_BACKUP_REMOTE.
FRESH_BID="$(sudo -u maid-engine --preserve-env=INST,MAID_SRC bash -c \
  'set -a; . "/etc/maid/${INST}.env"; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance "${INST}" --json' \
  | jq -r '.[]' | sort -r | head -1)"
export FRESH_BID
sudo -u maid-engine --preserve-env=INST,FRESH_BID,MAID_SRC bash -c \
  'set -a; . "/etc/maid/${INST}.env"; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance "${INST}" --backup-id "${FRESH_BID}"'

Step 11: communicate to players

# maid ops announce is functional but webhook-only (--channels is
# metadata, not routing). Export the webhook URL into this shell first.
export MAID_BRIDGES_WEBHOOK_URLS="discord:https://discord.com/api/webhooks/<id>/<token>"
sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --channels webhook --severity info \
  --message "Server restored after host failure. Data recovered to <BID's timestamp>. Apologies for the disruption — full details on the forum."
# Reach connected players in-game via the admin broadcast REST API:
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[NOTICE]","message":"Server restored after host failure — welcome back."}'
Post a longer-form retro in Discord/forum once the dust settles.

"If it goes wrong" branches

Provisioning the new box takes too long

  • Outside this runbook's scope, but: have a documented secondary hosting provider in escalation-contacts so you can fail over if the primary provider is the failure.

Off-host backup is unreachable

  • If you have a tertiary backup destination, use it.
  • If not, you are recovering from your LAST local backup, which died with the host. Communicate complete data loss to players. File an urgent follow-up to add a second off-host destination.

Verify fails on every BID you try

  • The backups themselves are corrupt. Pick the OLDEST you have and see how far back you can go. This is a multi-day data loss event. Be honest with players.

Engine fails to start on the new host

  • See engine log. Common causes:
  • PG version mismatch (dump from PG N, new host has PG N-1 or N+1). Install matching PG, or use pg_upgrade per Postgres docs.
  • Migration version mismatch (engine binary too new for the backed-up schema). Install the engine version recorded in the backup MANIFEST.json (code_version) first; once running, perform the upgrade in a normal maintenance window.

DNS propagation slow

  • Most players experience a few minutes of stale DNS resolving the dead IP. Communicate via Discord; provide the new IP as a workaround for power users.

Recovery

  • maid-admin doctor --phase runtime --instance <inst> all green on new host.
  • maid-admin status --json | jq .leader_lock shows the new host's engine as holder (PID matches the new maid-engine@<inst> process on this box, joined via pg_stat_activity where application_name LIKE 'maid-engine:%') and generation is incremented versus the inherited value.
  • /readyz returns 200.
  • DNS resolves to the new host.
  • Fresh local backup taken and verified on the new host.
  • Players connecting normally.

Post-incident

  • File a public-ish retro: what failed, what data was lost, what is changing to prevent recurrence.
  • Audit the DR procedure itself: where did this runbook fall short? Update it.
  • Update ./escalation-contacts.md.template "Last drilled" date.
  • Re-evaluate RPO/RTO: if the recovery point was older than acceptable, the backup schedule or off-host transport needs tuning.
  • File follow-up to drill RB18 quarterly so the next 3am-execution is faster than the first.

Escalation