Skip to content

Runbook:

Prerequisites

  • Interim maid CLI (MAID_SRC): there is no system-wide maid binary until M1.4 (install.sh leaves /opt/maid/venv empty), so the CLI runs from a source checkout. export MAID_SRC=/path/to/maid-checkout and build its venv once: sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv: /opt/maid/current/.uv/bin/uv). Commands below run "${MAID_SRC}/.venv/bin/maid" as the maid-engine user; inside bash -c blocks MAID_SRC is forwarded via --preserve-env=MAID_SRC.
  • Tools: maid-admin, ssh, psql, jq, journalctl.
  • Access: state the real privilege needed. Raw sudo systemctl/sudo journalctl on the instance units and sudo -u maid-engine …/sudo -u postgres … shells require root or an equivalently broad sudo grant — the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) only covers fixed maid-admin <verb> calls, and there is no maid-ops group in the packaging. List any SSH keys needed. If a step uses deploy/scripts/backup.sh, note that it is repo-only until M1.4 and add a MAID_SRC checkout prerequisite.
  • Env file: /etc/maid/<instance>.env readable.
  • Paths to know:
  • /var/lib/maid-engine/<instance>/ — engine state
  • /run/maid-engine/<instance>/ — runtime sockets, pidfiles
  • /var/log/maid/ — engine + audit logs
  • /var/lib/maid-backups/ — local backup staging
  • Audit access: tail -f /var/log/maid/ops-audit.jsonl works.
  • Offline status command: maid-admin status --instance <inst> (reports offline when the engine is down).
  • Escalation contacts: see ./escalation-contacts.md.template (template at ./escalation-contacts.md.template).

Symptoms

Summary

What the operator (or monitoring) sees when this fires.

Detection

  • Metric/log query that triggers this runbook.
  • Where it surfaces (Discord, email, doctor output, in-game).

Blast radius

  • Players affected: .
  • Data at risk: .
  • AI/external systems impact.

Diagnostic Steps

First 5 minutes (LITERAL commands)

# 1. Confirm
maid-admin status --instance <inst> || echo "engine offline"

# 2. Capture state (do this BEFORE poking anything)
sudo journalctl -u maid-engine@<inst>.service -n 200 > ./incident-$(date -u +%Y%m%dT%H%M%SZ).log
maid-admin status --instance <inst> --json > ./incident-status-$(date -u +%Y%m%dT%H%M%SZ).json 2>&1 || true

# 3. Quick action (the safest first mitigation; varies per runbook).
#    Notify connected players — `maid ops maintenance on` is an M9 stub and
#    does NOT gate logins (see ../deployment/player_comms.md). If the engine
#    is up, use the functional in-game broadcast:
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"investigating, ETA 15m"}'

Investigation

  • Step-by-step queries / log greps / maid-admin commands.
  • Decision tree if applicable.

Resolution Steps

Mitigation

  • Per-symptom remediation steps.

Recovery

  • How to verify the system is healthy again:
  • maid-admin doctor --phase runtime --instance <inst> returns all green.
  • /readyz returns 200.
  • maid-admin status --instance <inst> shows engine_state=running.
  • What to communicate to players: see ../deployment/player_comms.md.
  • Post the all-clear via the admin broadcast once verified (maid ops maintenance off is an M9 stub — no login gate to lift).

Post-incident

  • File a ticket: root cause, doc gaps, automation opportunities.
  • Update this runbook if 3am-you needed something it didn't say.
  • Update ./escalation-contacts.md.template if any contact changed.

Escalation

  • Solo path: stop the bleeding, defer fix, file ticket.
  • Hosting console URL: see ./escalation-contacts.md.template.
  • DNS registrar URL: see ./escalation-contacts.md.template.
  • Comms channel URL: see ./escalation-contacts.md.template.
  • Peer operator: see ./escalation-contacts.md.template.
  • If you cannot recover within 30 minutes, notify players. If the engine is up, use the in-game broadcast; if it is down, post to the webhook instead (maid ops maintenance is an M9 stub): 
    # engine up:
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"incident, ETA <ETA>"}'
# engine down: `maid ops announce` reads MAID_BRIDGES_WEBHOOK_URLS from its
# own process env — source the instance env with auto-export (a bare
# --preserve-env of a never-exported var forwards nothing):
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a;
   exec "${MAID_SRC}/.venv/bin/maid" ops announce --severity warn --message "incident, ETA <ETA>"'
    and post to the comms channel per ../deployment/player_comms.md.