Skip to content

Runbook: AI provider outage

Symptom

  • AI dialogue circuit breaker open; engine journal: AIProviderUnavailable: breaker open for <provider> repeating.
  • Spike of provider 5xx in maid_ai_provider_http_status_total{status="5xx"}.
  • Players report NPCs are silent / repeat canned fallback lines.
  • Provider's own status page reports incident (Anthropic / OpenAI / etc.).

Detection

  • maid_ai_breaker_state{provider="<p>"} == "open".
  • rate(maid_ai_provider_http_status_total{status=~"5..|429"}[5m]) > 0.5.
  • Provider status RSS / Discord notification.
  • Doctor: maid-admin doctor --phase runtime --instance <inst> ai_provider_reachable: warn.

Blast radius

  • Players affected: anyone interacting with AI NPCs. Core gameplay (look / move / get / drop / combat / chat between players) is unaffected.
  • Data at risk: none.
  • AI/external systems: by definition.

/readyz stays green. AI is not on the critical path for MAID v1. Do NOT declare maintenance for an AI-only outage; the right move is to disable AI cleanly and tell players AI features are temporarily off.

Prerequisites

  • Interim maid CLI (MAID_SRC): there is no system-wide maid binary until M1.4 (install.sh leaves /opt/maid/venv empty), so the CLI runs from a source checkout. export MAID_SRC=/path/to/maid-checkout and build its venv once: sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv: /opt/maid/current/.uv/bin/uv). Commands below run "${MAID_SRC}/.venv/bin/maid" as the maid-engine user; inside bash -c blocks MAID_SRC is forwarded via --preserve-env=MAID_SRC.
  • Tools: maid-admin, journalctl, curl (to probe provider).
  • Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units and sudo -u maid-engine … service-user shells, none of which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) grants (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root).
  • Env file: /etc/maid/<inst>.env — MAID_AI_*.
  • Paths to know:
  • Engine log: journalctl -u maid-engine@<inst>
  • Breaker state visible via maid-admin status --json.
  • Audit access: tail -f /var/log/maid/ops-audit.jsonl.
  • Escalation: see ./escalation-contacts.md.template.

First 5 minutes (LITERAL commands)

# 1. Confirm it's provider-side, not us
maid-admin status --instance <inst> --json \
  | jq '.ai.breaker_state, .ai.recent_errors'

# 2. Halt AI provider calls cleanly. `maid ops ai disable` is an M9 stub;
#    the FUNCTIONAL equivalent is the AI kill switch, which makes every
#    AI provider call raise ProviderUnavailable (tick loop unaffected).
#    `reason` is a positional argument; trip is DESTRUCTIVE (--confirm):
sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops kill-switch trip "<provider> outage" \
  --confirm --instance <inst>

# 3. Broadcast to players. `maid ops announce` is functional but
#    webhook-only (--channels is metadata, not routing); export the
#    webhook URL, and reach in-game players via the admin broadcast API.
export MAID_BRIDGES_WEBHOOK_URLS="discord:https://discord.com/api/webhooks/<id>/<token>"
sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --channels webhook --severity info \
  --message "AI NPC dialogue temporarily unavailable; gameplay continues. We'll bring it back when the provider recovers."
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[NOTICE]","message":"AI NPC dialogue temporarily unavailable; gameplay continues."}'

# 4. Capture state
sudo journalctl -u maid-engine@<inst>.service --since "-30 min" \
  | grep -iE 'ai_provider|breaker|429|503' \
  > ./incident-ai-$(date -u +%Y%m%dT%H%M%SZ).log

Investigation

  • Confirm it isn't auth / quota on our side: 
    # API-key-aware health probe (each provider has its own; example below)
source /etc/maid/<inst>.env
curl -sS -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $MAID_AI_<PROVIDER>_API_KEY" \
  "$MAID_AI_<PROVIDER>_HEALTH_URL"
# Expect: 200. 401/403 = our key. 429 = our quota. 5xx = provider.
  • Check provider status page (URLs in ./escalation-contacts.md.template).
  • Check maid-admin status --json | jq .ai.usage_today for quota proximity.

Mitigation

  • AI already disabled in step 2. Now wait for the provider to recover, OR fail over to a configured secondary provider: 
    # `maid ops ai set-provider`/`ai enable` are M9 stubs. To fail over to
# a secondary provider today: set it in the env file, restart to load
# it, then clear the kill switch.
sudo sed -i 's|^MAID_AI_DEFAULT_PROVIDER=.*|MAID_AI_DEFAULT_PROVIDER=<fallback>|' \
  /etc/maid/<inst>.env
sudo systemctl restart maid-engine@<inst>.service
sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops kill-switch reset --instance <inst>
  • If the outage is on OUR side (auth/quota), see Auth/Quota fixes below.

Auth/Quota fixes

  • 401/403: API key revoked/rotated. Update /etc/maid/<inst>.env with the new key, then: 
    # Env is read only at process start; `systemctl reload` sends an
# unhandled SIGHUP that kills the engine. Restart to load the new key,
# then clear the kill switch (functional; re-enables AI calls):
sudo systemctl restart maid-engine@<inst>.service
sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops kill-switch reset --instance <inst>
  • 429: quota / rate limit hit. Either wait for the rate window to reset, request a quota bump from the provider, or lower MAID_AI_DIALOGUE_GLOBAL_RATE_LIMIT_RPM in env and restart.

Recovery

When provider status page reports recovery: 

# Clear the AI kill switch (functional; re-enables AI provider calls).
sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops kill-switch reset --instance <inst>

# Watch the breaker transition open -> half-open -> closed
watch -n 5 'maid-admin status --instance <inst> --json \
  | jq ".ai.breaker_state, .ai.success_count_recent"'
- Breaker should transition open -> half-open after the cooldown, then half-open -> closed after N successful calls. - If it flips back to open, the provider isn't actually recovered; disable AI again and wait.

When breaker stays closed for ≥ 5 min and success rate is normal: 

# `maid ops announce` reads MAID_BRIDGES_WEBHOOK_URLS from its own process
# env; source the instance env with auto-export (a bare --preserve-env of a
# never-exported var forwards nothing):
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a;
   exec "${MAID_SRC}/.venv/bin/maid" ops announce --severity info \
     --message "AI NPC dialogue restored. Thanks for your patience."'

Post-incident

  • File ticket with: provider, outage start/end, our detection time, did the breaker behave correctly, did the announce reach all channels.
  • If failover to secondary provider was used and helpful, evaluate making that the default for the affected workload.
  • Update ./escalation-contacts.md.template if provider status URL or contact changed.

Escalation