Skip to content

Runbook: Planned maintenance window

Symptom

You are running a planned maintenance window (deploy, schema migration, host move, DB tuning, etc.). This runbook is the procedure for the window itself, with pre- and post-window comms cadence and per-protocol verification.

For unplanned outages (incident-driven maintenance), the relevant incident runbook will tell you to declare maintenance and refer back here for the comms cadence and verification.

Detection

Not detection — operator decision following a deploy plan or change ticket.

Blast radius

  • Players affected: ALL on this instance for the announced window.
  • Data at risk: depends on the change. Maintain a verified backup before entering the window (see Prerequisites).
  • AI/external systems: same.

Prerequisites

  • Interim maid CLI (MAID_SRC): there is no system-wide maid binary until M1.4 (install.sh leaves /opt/maid/venv empty), so the CLI runs from a source checkout. export MAID_SRC=/path/to/maid-checkout and build its venv once: sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv: /opt/maid/current/.uv/bin/uv). Commands below run "${MAID_SRC}/.venv/bin/maid" as the maid-engine user; inside bash -c blocks MAID_SRC is forwarded via --preserve-env=MAID_SRC.
  • Tools: maid-admin, systemctl, journalctl, psql, plus whatever the change itself requires.
  • Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units and sudo -u maid-engine … service-user shells, none of which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) grants (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root).
  • Source checkout (MAID_SRC): deploy/scripts/backup.sh is repo-only and is not staged under /opt/maid/current until M1.4, so export MAID_SRC=/path/to/maid-checkout before running the backup steps below; they invoke "${MAID_SRC}/deploy/scripts/backup.sh" and forward MAID_SRC through sudo with --preserve-env=MAID_SRC.
  • Env file: /etc/maid/<inst>.env.
  • A verified backup less than 1 h old. If not, take one and verify it fail-closed: 
    # NOTE: on a stock host this fails 203/EXEC until M1.4 stages
# bin/backup.sh; until then run ./deploy/scripts/backup.sh full from a
# checkout instead (see backup.md "Manual backup (interim)").
sudo systemctl start maid-backup@<inst>.service
sudo journalctl -u maid-backup@<inst>.service -f   # wait for SUCCESS, then Ctrl-C
# `list --json` is a JSON array of id STRINGS (not objects); ids begin
# with a UTC timestamp, so `jq -r '.[]' | sort -r | head -1` is newest.
# backup.sh reads MAID_DEPLOY_BACKUP_REMOTE from its OWN env; source the
# instance env with auto-export inside the maid-engine shell (a bare
# sudo does not load EnvironmentFile=).
BID="$(sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
        'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance <inst> --json' \
        | jq -r '.[]' | sort -r | head -1)"
test -n "$BID" || { echo "FAIL: no backup found"; exit 1; }
export BID
# verify re-downloads and sha256-checks every file; non-zero on any
# mismatch/missing sidecar. Gate the window on its exit code.
sudo -u maid-engine --preserve-env=BID,MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance <inst> --backup-id "${BID}"' \
  || { echo "FAIL: $BID did not verify — do NOT start the window"; exit 1; }
# Age gate: the id timestamp (first field) must be < 1h old.
TS="${BID%%-*}"
AGE=$(( $(date -u +%s) - $(date -u -d "${TS:0:4}-${TS:4:2}-${TS:6:2} ${TS:9:2}:${TS:11:2}:${TS:13:2} UTC" +%s) ))
test "$AGE" -lt 3600 || { echo "FAIL: newest backup is $((AGE/60))m old (> 60m)"; exit 1; }
    See ./backup_failed.md if the backup fails.
  • A rollback plan: pre-document which runbook you'd use if the change fails (typically ./rollback.md and/or ./restore.md).
  • Player-comms templates: see ../deployment/player_comms.md.
  • Escalation: see ./escalation-contacts.md.template.

Pre-window comms cadence

maid ops announce is functional, but it is webhook-only: it POSTs to the endpoints in MAID_BRIDGES_WEBHOOK_URLS. --channels is audit metadata, not routing — it does not reach in-game players, Discord-by-name, or a status page. Export the webhook target into the announcing shell once; the --preserve-env forwards it across sudo:

export MAID_BRIDGES_WEBHOOK_URLS="discord:https://discord.com/api/webhooks/<id>/<token>"

To reach connected players in-game, use the admin broadcast REST API (shown at T-5 min) — announce cannot do it.

T-24 h: announce (webhook)

sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --channels webhook --severity info \
  --message "Scheduled maintenance on <DATE> <START_TIME>-<END_TIME> <TZ>. Expected downtime: <N> minutes. Reason: <REASON>."
announce prints a per-endpoint OK/ERR line and exits non-zero if any webhook POST failed — check that before trusting the announcement.

T-1 h: remind (webhook)

sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --channels webhook --severity info \
  --message "Reminder: maintenance in 1 hour. Expected downtime ~<N>m. Save your progress and find a safe room."

T-15 min: final warning (webhook)

sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --channels webhook --severity warn \
  --message "Maintenance in 15 minutes. Server will shut down briefly. We will return at <END_TIME> <TZ>."

T-5 min: in-game broadcast (reaches all connected sessions)

# announce does NOT reach players; the admin broadcast REST API does.
# maid ops broadcast is an M9 stub, so call the endpoint directly.
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"Maintenance in 5 minutes — final warning."}'

Entering the window

# 1. Confirm a verified backup exists < 1h old (fail-closed; see
#    Prerequisites for the full BID + verify + age-gate sequence —
#    `list --json` returns id STRINGS, so do NOT `jq '.[0]|{id,ts}'`).
BID="$(sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
        'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance <inst> --json' \
        | jq -r '.[]' | sort -r | head -1)"
test -n "$BID" || { echo "FAIL: no backup"; exit 1; }
export BID
sudo -u maid-engine --preserve-env=BID,MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance <inst> --backup-id "${BID}"' \
  || { echo "FAIL: backup did not verify — abort"; exit 1; }

# 2. Warn players. maid ops maintenance is an M9 stub (no login gate is
#    enforced), so there is no working "maintenance mode" toggle — warn
#    via webhook announce + admin broadcast, then rely on the stop below.
sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --channels webhook --severity warn --message "Maintenance starting now."
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"Maintenance starting now; the server will restart shortly."}'

# 3. (maid ops drain is an M9 stub — there is no graceful drain today.)

# 4. Stop the engine (if the change requires it; many do not)
sudo systemctl stop maid-engine@<inst>.service

# 5. Perform the actual change (deploy, migration, DB tuning, host move, ...)
#    Document each command in the change ticket as you go.

# 6. Start the engine
sudo systemctl start maid-engine@<inst>.service

Per-protocol verification

After the engine reports engine ready in journal, verify EACH protocol the deployment serves. Do NOT skip protocols you "don't think matter" — players use all of them.

Telnet — ANSI banner sample

# Expect: ANSI-colored login banner + version line + prompt.
(printf '\r\n'; sleep 1) | nc -w 2 localhost 4000 | head -20
# Spot-check: ANSI escapes (`\x1b[`) render; banner is current version.
What you should see (literal expectation): - \x1b[ sequences interleaved with the banner. - Server version line matches the deployed version (maid-admin status --json | jq .build). - A login prompt (e.g., Login: or Username:).

WebSocket — structured event sample

# Use any ws client (websocat / wscat). Example with websocat:
echo '{"type":"ping"}' | websocat -n1 --jsonl ws://localhost:8080/ws
# Expect: a JSON pong/welcome event back.
What you should see: - A JSON object (not HTML — confirms the WS upgrade succeeded). - type field is a known event from the GMCP/WS protocol.

Admin UI — modal sample

# Manually: open https://<host>/admin-ui/ in a browser.
Verify: - Login modal renders without console errors. - Build version in the UI footer matches maid-admin status .build. - "Server status" panel shows engine_state=running.

HTTP probes

curl -fsS http://localhost:9090/healthz   # liveness
curl -fsS http://localhost:9090/readyz    # readiness
curl -fsS http://localhost:9090/livez     # process alive
All three return 200 with ok in body.

Doctor — last gate

maid-admin doctor --phase runtime --instance <inst>
# Every check must be pass. Do NOT exit maintenance until all green.

Exiting the window

# 1. (maid ops maintenance off is an M9 stub — nothing to toggle.
#    The engine start above already restored normal player logins.)

# 2. Verify players can connect (canary)
#    Hop on with your own account; confirm look / move / chat work.

# 3. Take a fresh backup so the post-change state is the new restore baseline
# NOTE: on a stock host this fails 203/EXEC until M1.4 stages bin/backup.sh;
# until then run ./deploy/scripts/backup.sh full from a checkout instead.
sudo systemctl start maid-backup@<inst>.service
sudo journalctl -u maid-backup@<inst>.service -f   # wait for SUCCESS, then Ctrl-C
BID="$(sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
        'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance <inst> --json' \
        | jq -r '.[]' | sort -r | head -1)"
test -n "$BID" || { echo "FAIL: no backup"; exit 1; }
export BID
sudo -u maid-engine --preserve-env=BID,MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance <inst> --backup-id "${BID}"' \
  || { echo "FAIL: post-change backup did not verify"; exit 1; }

Post-window comms cadence

Immediate: "we're back"

sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --channels webhook --severity info \
  --message "Maintenance complete. Server is back online. Thanks for your patience!"
# Also tell connected players in-game:
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"Maintenance complete — welcome back!"}'

T+1 h: change summary (only if user-visible)

Post to discord/forum the user-visible changes (new content, bugfixes, known regressions). Template lives in ../deployment/player_comms.md.

"If it goes wrong" branches

Change fails mid-window

Window runs over

  • Extend the announce: 
    sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --channels webhook --severity warn \
  --message "Maintenance extended to <NEW_END_TIME> <TZ>. Sorry for the delay."
  • Reassess at every 30-minute mark whether to keep going or rollback.

Verification fails post-deploy

  • Do NOT exit maintenance with a failing protocol. Each broken protocol is its own incident:
  • Telnet broken → engine log + check listener bound (ss -tlnp).
  • WS broken → check the WS endpoint + reverse proxy if used.
  • Admin UI broken → re-build / re-deploy frontend per the deploy guide.

Recovery / verification (planned exit)

  • Doctor all-green BEFORE comms.
  • All four protocols (telnet, ws, admin UI, HTTP probes) verified.
  • Fresh backup taken and verified.
  • Players connecting normally (canary login).
  • "Back online" comms posted.

Post-incident (per-window retro)

  • File a window report: planned vs actual duration, what went right, what went wrong, comms timing actually hit, any rollback used.
  • Update this runbook with anything that didn't match the plan.

Escalation