Skip to content

Runbook — running chaos tests

Audience: SRE on a chaos game-day, or an operator vetting a brand-new host before declaring it "in service."

Owner: SRE rotation. Chaos runs must be announced ≥24h ahead in the operator channel and recorded in the post-run log.

When to run

Trigger What to run Where
New host being added to fleet Full catalogue (#1–#12) Staging clone of new host, not the live host
Quarterly resilience review Full catalogue Dedicated chaos host
After a real incident The scenario(s) matching the incident's failure class Staging
After changing the persistence/migration/backup code paths #1, #7, #9, #10 minimum Staging
Before a release that bumps MAID_DRAIN_TIMEOUT_S or MAID_TICK_RATE #1 + #11 Staging

Never run any chaos scenario on a host that is currently serving real players. The catalogue assumes the operator has at minimum a staging clone and a way to redirect player traffic away first.

Prerequisites

  1. Opt-in gate set. Every test in tests/chaos/ is no-op unless MAID_CHAOS_ENABLE=1 is exported:
export MAID_CHAOS_ENABLE=1

  1. Target host identified. Use a host whose hostname starts with chaos- or staging-. The operator handbook reserves these prefixes for non-production.

  2. Backup taken. Run a full backup and verify it (deploy/scripts/backup.sh verify) before injecting any state-mutating chaos.

  3. Alert pause arranged. Silence the noisy alerts that you expect to fire (the chaos run itself will fire them and you don't want the on-call drowning in pages from your own test):

# Example for an Alertmanager-based stack:
amtool silence add alertname=MaidHealthDegraded \
    --duration 2h --comment "chaos run @qworg"
  1. Game-day buddy. Pair with another operator who is not running the chaos. They watch dashboards and call abort if real traffic is affected.

Execution

Each scenario has a documented entry in chaos catalogue with five fields:

  • Scenario — what to break
  • Expected behavior — what should happen
  • Observable signals — what to watch (metric / log / event)
  • Recovery procedure — how to put the world back
  • Validation script — executable harness (✅) or manual steps (🟡)

Workflow for a single scenario:

# 1. Pre-run baseline.
curl -s "http://${HOST}:9090/api/v1/query?query=maid_health_status" | jq

# 2. Run the validation script (or follow manual steps).
MAID_CHAOS_ENABLE=1 bash tests/chaos/test_kill_during_save.sh

# 3. Check the observable signals match the expected behavior.
journalctl -u maid-engine@chaos.service -n 200 --no-pager | tail -50

# 4. Run the recovery procedure if the script didn't restore state.

# 5. Re-baseline. The metric you watched in step 1 must return to
#    its starting value within the runbook's recovery time (RTO).

# 6. Record the result in the chaos log (see "Recording" below).

For the full catalogue run, iterate through scenarios #1–#12 in order. Stop and triage at the first scenario whose expected behavior did not occur — that is a real resilience defect.

Recording

Append a line per scenario to /var/log/maid/chaos.log:

2025-01-15T14:22:01Z scenario=1 operator=qworg buddy=alex result=pass notes="drain in 8.2s"
2025-01-15T14:31:44Z scenario=2 operator=qworg buddy=alex result=fail notes="OOM-killer fired but ready-file appeared before persistence caught up; filed ISSUE-1234"

Failures get an issue tracker entry within 24h. Passes get archived into the quarterly resilience report.

Recovery if things go sideways

Each scenario in the catalogue has its own recovery procedure linked out to the appropriate runbook. The generic fallbacks, in order:

  1. systemctl stop maid-engine@chaos.service — quench the engine first.
  2. nft -a list ruleset then nft delete rule ... handle <h> — remove any chaos firewall rules.
  3. umount /var/lib/maid/chaos/state (if you mounted a tmpfs for #3 / #10) and restore the underlying mount.
  4. pg_advisory_unlock_all() from a fresh psql session — drop any stale advisory locks from #8.
  5. chronyc -a makestep — resync clocks after #6.
  6. pkill stress-ng — kill any leftover load generators from #11.
  7. Re-mount, restart, verify integrity: 
    systemctl start maid-engine@chaos.service
# The engine runs a persistence integrity check on startup; for an
# explicit re-check, run `@persistence check` from an in-game admin
# session (there is no shell `maid persistence` command).

If the host is unrecoverable, treat it as a host-loss event and follow RB18 — host loss / DR: destroy the staging host and rebuild from the bare-metal deployment guide.

Don'ts

  • Don't run chaos against the live cluster, ever.
  • Don't disable the MAID_CHAOS_ENABLE gate "to save time" — it exists because production CI will eventually run these tests and we don't want them firing then.
  • Don't silence chaos alerts longer than the planned run window.
  • Don't chain destructive scenarios without recovering between each — you'll lose ability to attribute failures.
  • Don't skip the recording step — the audit trail is the resilience program's deliverable.

See also