MAID Runbooks¶

PRINT THIS. If the host is unreachable you cannot read this in vim. The bundle tarball ships with PDFs; print at least the must-print list below and keep them in the rack.

Must print¶

Reprint cadence: write the print date on the cover sheet (Last printed: YYYY-MM-DD). If the date is older than 30 days OR the runbook source has been updated in git log since you printed it, reprint before the next on-call shift. The bundle CI (runbooks-bundle.yml) attaches a fresh PDF set to every release, so the safest cadence is "reprint on every MAID upgrade."

Index by symptom¶

Symptom	Runbook
`/readyz` returns 503; engine journal full of `asyncpg.exceptions.ConnectionDoesNotExistError`.	db_down.md
AI dialogue queue depth grows; NPC responses get slow or empty. `/readyz` typically stays green.	redis_down.md
Engine refuses writes; players see "the world resists your change" errors; disk-monitor alert fires.	disk_full.md
`systemctl status maid-engine@<inst>` shows recent restarts; OOM-killed in dmesg.	oom_loop.md
`maid_persistence_save_queue_oldest_age_seconds` exceeds 2× `MAID_PERSISTENCE_SAVE_INTERVAL`.	save_queue_growing.md
`maid_backup_last_success_timestamp_seconds` stale > 25 h OR `maid-backup@<inst>.service` exited non-zero.	backup_failed.md
You need to restore from a verified backup (sent here by another runbook or by an operator drill).	restore.md
`maid-engine@<inst>.service` refuses to start; journal says `MigrationFailure`.	failed_migration.md
A newly deployed release is misbehaving and you need to swap `/opt/maid/current` back.	rollback.md
AI dialogue breaker open; spike of provider 5xx / 429. `/readyz` stays green.	ai_provider_outage.md
Second `maid-engine@<inst>` start refused with `LeaderLockHeld(holder_pid=...)`.	two_instances_detected.md
Planned maintenance window (deploy, migration, host move). Comms cadence + per-protocol verification.	maintenance_window.md
You need a host-level firewall change RIGHT NOW (DDoS, brute force, suspected intrusion).	RB17_firewall_fallback.md
The MAID host is gone or unrecoverable. Full disaster-recovery rebuild on a new box.	RB18_host_loss_dr.md
Engine fails to spawn threads or open files; TasksMax / MemoryMax / fd exhaustion.	RB19_resource_exhaustion.md
`install.sh` aborts with `python 3.12 not found` or `uv: command not found`.	RB22_python_uv.md
You received a GDPR Article 17 ("right to erasure") request and need to remove a player's data from the live DB.	player_data_erasure.md
Scheduled or compromise rotation of DB / AI / backup-transport SSH / admin-API / `MAID_ADMIN_SECRET_KEY` secrets.	secret_rotation.md

Format¶

Every runbook follows _template.md:

Symptom → Detection → Blast radius → Prerequisites → First 5 minutes (LITERAL commands) → Investigation → Mitigation → Recovery → Post-incident → Escalation.

Cross-runbook references use relative links. Plan revision IDs are embedded as HTML comments for traceability.

Escalation contacts¶

A template lives at ./escalation-contacts.md.template. Copy it to /etc/maid/escalation-contacts.md.template on each host (mode 0640 root:maid-admin) and fill it in. Do NOT commit the filled-in file. Every runbook references this file by name.

Bundle¶

On every release the CI runs .github/workflows/runbooks-bundle.yml and attaches runbooks-bundle.tar.gz to the GitHub release with markdown + PDF + MANIFEST.txt + escalation-contacts.md.template.

The intent is that an operator can wget the tarball from a fresh box or unzip from a printed-and-scanned QR-coded backup, and have the runbooks on hand without internet access.

Maintainer note — pending CI workflows. Three workflow files in this bundle cannot be pushed by the Copilot service account due to the OAuth scope on .github/workflows/*.yml. They are staged in the agent session-state and need a human maintainer with workflow scope to commit them: check-no-leaked-contacts.yml (filled-in escalation-contacts.md guard), backup-integration.yml (backup script integration tests), and runbooks-bundle.yml (pandoc → PDF bundle, this section's release artifact). Until merged, the bundle attached to the GitHub release will be missing.