Chaos test catalogue¶
Audience: SREs and on-call operators verifying that MAID's production resilience claims hold under real failure injection.
Style: Litmus-style scenarios — one row per failure class, declarative description of what to break, what should happen, what to look at to confirm it happened, how to put the world back, and a pointer to an executable validation script.
Status legend: ✅ implemented script · 🟡 stub / manual steps only · ⏳ TODO.
Running these is gated by
MAID_CHAOS_ENABLE=1. See chaos-test operational runbook for the standing-order on when (and on what hosts) these may be exercised.
Scenario catalogue¶
1. Process kill (SIGTERM, SIGKILL)¶
Scenario. Kill the running maid service process mid-tick, twice:
once with SIGTERM (graceful drain expected) and once with SIGKILL
(no chance to flush) immediately after a save was issued.
Expected behavior.
- SIGTERM: drain manager catches the signal, completes the current
tick, flushes the save queue, closes player sessions with the
"server restarting" MOTD, and exits 0 within MAID_DRAIN_TIMEOUT_S.
- SIGKILL: process exits non-zero immediately. On restart, the
durability layer must replay/repair from the last consistent
checkpoint without losing committed writes (RPO ≤ tick interval).
Observable signals.
- journalctl -u maid-engine@<inst>.service shows the drain log line for
SIGTERM; no drain line for SIGKILL.
- maid_drain_seconds histogram (Prometheus) gets a sample for the
SIGTERM run.
- After restart, maid_persistence_integrity_check_failures_total
must stay at 0.
- For SIGKILL: maid_save_queue_replay_entries_total should be
non-zero on the next startup.
Recovery procedure.
1. systemctl start maid-engine@<inst>.service
2. @persistence check (in-game admin command) — must return clean.
3. If integrity check reports inconsistency, follow
runbooks/restore.md.
Validation script. tests/chaos/test_kill_during_save.sh ✅
2. OOM (cgroup memory limit hit)¶
Scenario. Lower the systemd unit's MemoryMax= to a value below
current RSS (e.g. MemoryMax=256M on an instance using 600M) and
systemctl daemon-reload && systemctl restart. Then drive load
(open 50 telnet sessions) until the kernel OOM-killer fires.
Expected behavior.
- kernel OOM message in dmesg; systemd restarts the unit per
Restart=on-failure with RestartSec=10s.
- ready-file signal is not emitted until persistence catches up.
- Players see the "server restarting" MOTD; sessions reconnect cleanly.
Observable signals.
- journalctl -k | grep -i oom-kill — at least one entry.
- node_systemd_unit_state{name="maid-engine@<inst>.service"} flaps from
active → activating → active in Prometheus.
- maid_startup_seconds histogram gets a fresh sample.
Recovery procedure.
1. Raise MemoryMax= back to the documented baseline (grep -r
MemoryMax packaging/systemd/).
2. systemctl daemon-reload && systemctl restart maid-engine@<inst>.service.
3. Inspect oom_loop runbook if the
service immediately re-OOMs.
Validation script. 🟡 manual; see runbook.
3. Disk full (state, log, runtime dirs)¶
Scenario. Three sub-tests, one per critical mount:
state-dir: fill/var/lib/maid-engine/<inst>to 100%.log-dir: fill/var/log/maid-engine/<inst>to 100%.runtime-dir: fill/run/maid-engine/<inst>to 100% (tmpfs).
Use dd if=/dev/zero of=<dir>/.chaos-pad bs=1M to grow until ENOSPC.
Expected behavior.
- state-dir: persistence layer halts writes, refuses to drop data;
maid_save_queue_depth grows until alert threshold.
- log-dir: log shipper rotates and drops oldest; engine remains
serving.
- runtime-dir: ops socket and lock files are unaffected if
pre-allocated; new sockets fail loudly.
Observable signals.
- df -h /var/lib/maid-engine/<inst> → 100%.
- maid_persistence_writes_blocked_total → non-zero.
- no automatic Slack/Discord alert fires — the bridges WebhookChannel
only sends on a manual maid ops announce. If the observability
webhook bridge is enabled, the elevated metrics appear in its next
periodic snapshot.
Recovery procedure.
1. rm <dir>/.chaos-pad (or whatever you used to fill).
2. @persistence flush (in-game admin command) to drain the queue.
3. Cross-check with disk_full runbook.
Validation script. tests/chaos/test_disk_full_during_backup.py ✅ (focused on the backup path; full matrix is per-mount manual).
4. Network partition (egress block to Anthropic API)¶
Scenario. nft add rule inet filter output ip daddr api.anthropic.com drop
(or equivalent for whichever AI provider is MAID_AI_DEFAULT_PROVIDER).
Drive NPC dialogue load.
Expected behavior.
- AI provider client times out after MAID_AI_HTTP_TIMEOUT_S.
- OffTickLLMQueue retries with exponential backoff, then trips the
AI-provider circuit breaker.
- NPC dialogue degrades to canned fallback responses; tick loop
unaffected.
- maid_ai_provider_circuit_state{provider="anthropic"} flips to
open.
Observable signals.
- maid_ai_provider_request_errors_total → climbs.
- no automatic Slack/Discord alert is emitted; wire the
maid_ai_provider_* metrics into external alerting (e.g. Prometheus
Alertmanager) if you want to be paged.
- player telemetry: maid_npc_dialogue_fallback_responses_total
rises.
Recovery procedure.
1. nft delete rule inet filter output handle <handle> to remove
the egress block (nft -a list ruleset to find handle).
2. Circuit breaker recloses within MAID_AI_CIRCUIT_RECOVERY_S.
3. See runbooks/ai_provider_outage.md.
Validation script. 🟡
5. Slow disk (iolimit dd injection)¶
Scenario. Use systemd-run --slice=maid.slice -p IOReadBandwidthMax="/dev/sda 1M"
to throttle disk I/O to 1 MB/s; or dd if=/dev/zero of=/var/lib/maid-engine/<inst>/.chaos-load bs=1M count=10000 oflag=direct
in parallel with workload to saturate the device.
Expected behavior.
- maid_tick_duration_seconds histogram p99 climbs past tick budget.
- Tick-budget watchdog fires tick_overrun event.
- Saves coalesce (batched-write path); maid_save_queue_depth
grows but no data is dropped.
Observable signals.
- iostat -x 1 shows >90% %util.
- maid_tick_overruns_total counter increments.
Recovery procedure.
1. Stop the dd load, remove .chaos-load file.
2. Restore I/O limits: systemctl set-property maid.slice IOReadBandwidthMax=.
3. Save queue drains naturally.
Validation script. 🟡
6. Clock skew (chronyd offset)¶
Scenario. sudo chronyc -a 'makestep -300 1' (jumps clock back
5 min) — or set a static offset via timedatectl set-time. Hit the
engine with player auth + JWT-bearing admin requests.
Expected behavior.
- JWT exp/iat validation uses a leeway ≥ 60s; small offsets do
not invalidate tokens.
- Offsets > leeway: admin/REST requests return 401; player
sessions stay open (telnet doesn't validate clock).
- Tick scheduler uses time.monotonic() and is unaffected.
- Audit log entries record their host clock; replay reconstructs
intent ordering by sequence number, not timestamp.
Observable signals.
- chronyc tracking reports the offset.
- maid_auth_token_validation_failures_total{reason="expired"} spikes.
Recovery procedure.
1. sudo chronyc -a makestep resyncs.
2. systemctl restart chronyd if it refuses.
3. No engine restart required.
Validation script. 🟡
7. File corruption (random byte flip in state file)¶
⏳ PLANNED — not exercisable as written. MAID has no local
state.dbfile. Durable entity state is persisted to PostgreSQL viaPostgresDocumentStore(maid_engine.storage.document_store);/var/lib/maid-engine/<inst>holds content/staging data, not a single mutable state file to byte-flip. A meaningful corruption scenario must instead target the PostgreSQL data files (or a staged backup archive) and is not yet scripted. The expectations below are the intended contract once such a harness exists — do not treat them as verified current behavior.
Scenario (planned). Stop the engine, corrupt a byte in the PostgreSQL data directory backing the instance's database (or in a staged backup archive), then restart / run a restore.
Expected behavior (planned).
- Startup runs the persistence integrity check
(maid_engine.persistence.integrity.IntegrityChecker; also invokable via
the @persistence check in-game admin command) and detects the mismatch.
- Engine refuses to enter ready state; ready-file is not emitted.
- There is no built-in engine webhook/bridge event for corruption. Wire
the integrity-failure signal into external alerting (e.g. Prometheus
Alertmanager) if you want to be paged; player notice is a manual
maid ops announce.
- Operator runbook directs a restore from the last good backup.
Observable signals.
- journalctl -u maid-engine@<inst>.service shows the integrity-check
failure.
- node_systemd_unit_state stays in activating.
Recovery procedure.
1. systemctl stop maid-engine@<inst>.service.
2. Follow runbooks/restore.md to
restore from deploy/scripts/restore.sh staging →
deploy/scripts/restore.sh promote.
3. @persistence check (in-game admin command) must report clean before starting.
Validation script. ⏳ TODO — harness not yet written.
8. DB advisory lock held by zombie process¶
Scenario. Acquire MAID's migration advisory lock from a separate
psql session: SELECT pg_advisory_lock(<MAID_MIGRATION_LOCK_KEY>);,
then ^Z the psql to background it without releasing. Run
maid db migrate.
Expected behavior.
- Migration runner times out waiting for the lock after
MAID_MIGRATION_LOCK_TIMEOUT_S and exits non-zero with a clear
diagnostic naming the lock holder (pid + application_name from
pg_stat_activity).
- No partial migration is applied.
Observable signals.
- pg_stat_activity shows the zombie psql holding the lock.
- maid_migration_lock_wait_timeouts_total → 1.
- migration-runner log line names the blocking PID.
Recovery procedure.
1. fg the psql and \q (or kill <pid>).
2. Re-run maid db migrate.
3. See runbooks/failed_migration.md.
Validation script. 🟡
9. Crash during migration apply¶
Scenario. Insert a long-running statement into a test migration
(e.g. SELECT pg_sleep(60); mid-script). While that statement is
executing, kill -9 the maid db migrate process.
Expected behavior.
- Migration framework uses transactional advisory locking +
checkpoint table; the half-applied migration is marked
in_progress.
- Subsequent maid db migrate refuses to advance until operator
runs maid db repair or maid db rollback --namespace <ns>
(--namespace is required; add --dry-run first to preview).
- No checkpoint-based steps are skipped.
Observable signals.
- maid_migrations table has a row with status='in_progress' and
no completed_at.
- maid db status flags the in-progress row.
Recovery procedure.
1. maid db status to identify the stuck migration.
2. maid db repair --clear-pending (manual decision) or
maid db rollback --namespace <affected-ns> [--steps N]
(--namespace is required; rehearse with --dry-run first).
3. See runbooks/failed_migration.md.
Validation script. 🟡
10. Crash during backup¶
Scenario. While deploy/scripts/backup.sh full --instance <inst> is mid-stream
(during pg_basebackup / the content-tar step), kill -9 the backup
process.
Expected behavior.
- backup.sh builds each backup in a local staging directory
${MAID_BACKUP_LOCAL_DIR}/<BACKUP_ID> and only uploads to the
transport (transport_put) after the stage is fully built and
its MANIFEST.json written. A kill -9 mid-stage therefore leaves
an incomplete stage dir locally and uploads nothing — the
transport gains no new <BACKUP_ID>, so no partial backup is ever
promoted to the canonical destination.
- The next backup.sh full run calls prune_local_staging, which
keeps only the newest --keep N stage dirs by mtime; the abandoned
incomplete stage is reclaimed once it falls outside that window.
(There is no .partial suffix and no fixed "older than 1h" sweep.)
- maid_backup_last_success_timestamp_seconds{instance} does not
advance, and maid_backup_last_status{instance} keeps its prior
value — backup.sh does not write a failure metric itself, and
SIGKILL bypasses any exit handling.
Observable signals.
- ls "${MAID_BACKUP_LOCAL_DIR}/" shows the incomplete
<BACKUP_ID> stage dir (format <UTCstamp>-<instance>-<hex8>,
e.g. 20250115T093000Z-prod-1a2b3c4d); the transport destination
has no matching <BACKUP_ID>.
- journalctl -t maid-backup shows the killed invocation (manual runs
log under the maid-backup tag, not a systemd unit).
- The staleness of maid_backup_last_success_timestamp_seconds grows.
There is no maid_backup_failures_total counter; the only
automated failure metric (maid_backup_last_status=0 +
maid_backup_last_failure_timestamp_seconds) is written by the
OnFailure=maid-backup-failure@ hook, which requires the
maid-backup@<inst>.service unit — currently non-functional
(ExecStart @@MAID_HOME@@/bin/backup.sh is not installed → status
203/EXEC) — so it does not fire for manual runs.
Recovery procedure.
1. Remove the abandoned stage dir (or let the next run's
prune_local_staging reclaim it):
rm -rf "${MAID_BACKUP_LOCAL_DIR}/<BACKUP_ID>".
2. Re-run the backup manually from a repo checkout with the instance
env exported (the packaged timer/service is non-functional — see
above). backup.sh full requires --instance <inst> and reads
MAID_DEPLOY_BACKUP_REMOTE from its own process env, so source the
instance env first:
sudo -u maid-engine bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec ./deploy/scripts/backup.sh full --instance <inst>'
(run from the checkout root).
3. See runbooks/backup_failed.md.
Validation script. tests/chaos/test_disk_full_during_backup.py ✅ (covers the related disk-full sub-scenario; pure-crash variant is 🟡).
11. Stuck tick (system slowdown >5x tick budget)¶
Scenario. stress-ng --cpu $(nproc) --cpu-load 95 --timeout 60s
on the host while the engine is serving load.
Expected behavior.
- Tick scheduler logs tick_overrun warnings.
- After N consecutive overruns > 5× budget, the engine declares
itself degraded and sheds load (refuses new sessions with a
busy MOTD). There is no built-in engine webhook/bridge event for
this — wire the metrics below into external alerting (e.g. Prometheus
Alertmanager) if you want to be paged.
- It does not silently fall behind on saves; the save queue
uses backpressure (max_queue_depth) and signals upstream.
Observable signals.
- maid_tick_overruns_total rapidly increments.
- maid_health_status{check="tick"} → degraded.
- node_load1 > 2× nproc.
- no automatic alert fires; page on maid_health_status{check="tick"}
and maid_tick_overruns_total via external alerting.
Recovery procedure.
1. Kill the noise (pkill stress-ng).
2. Engine self-recovers; maid_health_status flips back to ok
once tick_lag stays under budget for ≥30s.
3. See runbooks/RB19_resource_exhaustion.md.
Validation script. 🟡
12. Webhook endpoint 503ing¶
Scenario. Point MAID_BRIDGES_WEBHOOK_URLS at a sink that always
returns 503 (e.g. a python3 -m http.server proxied through a stub).
Trigger a send with maid ops announce.
Expected behavior.
- The webhook channel POSTs, sees the 503 (a 5xx), retries exactly
once, then gives up — there is no exponential backoff and no
configurable retry cap.
- After the single retry fails, the channel logs one ERROR (not a
flood), increments maid_webhook_send_errors_total{format=...}, and —
because the fan-out runs only inside the announce CLI process — the
running engine is completely unaffected.
- No process restart, no tick impact, no save delay.
Observable signals.
- maid_webhook_send_errors_total{format="discord"} (or the
configured format) increments by 1 per failed announce.
- single ERROR line per failed endpoint, not per retry.
Recovery procedure.
1. Restore the webhook endpoint (or update MAID_BRIDGES_WEBHOOK_URLS
to a working URL in the environment used by announce).
2. Re-run maid ops announce — the CLI reads the env at invocation, so
no engine reload or restart is required.
3. See Discord webhooks guide → Troubleshoot.
Validation script. 🟡
Adding new scenarios¶
When you observe a novel failure mode in production, add an entry here before writing the validation script — the catalogue is the contract; the script proves it. Use the existing scenarios as templates and follow the same Scenario / Expected / Observable / Recovery / Validation-script shape.
See also:
- Chaos test operational runbook — when and how to run these.
- SLO / RPO / RTO targets — the numbers each scenario is implicitly testing.
- Runbook index — per-symptom recovery procedures referenced above.