Skip to content

Runbook: Redis down / unreachable

Prerequisites

  • Interim maid CLI (MAID_SRC): there is no system-wide maid binary until M1.4 (install.sh leaves /opt/maid/venv empty), so the CLI runs from a source checkout. export MAID_SRC=/path/to/maid-checkout and build its venv once: sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv: /opt/maid/current/.uv/bin/uv). Commands below run "${MAID_SRC}/.venv/bin/maid" as the maid-engine user; inside bash -c blocks MAID_SRC is forwarded via --preserve-env=MAID_SRC.
  • Tools: maid-admin, redis-cli, systemctl, journalctl, df.
  • Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units, which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) does not grant (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root).
  • Env file: /etc/maid/<inst>.env — MAID_REDIS_URL.
  • Paths to know:
  • /var/lib/redis/ — RDB / AOF data
  • /var/log/redis/ or journalctl -u redis
  • /etc/redis/redis.conf (Debian) or /etc/redis.conf (RHEL)
  • Audit access: tail -f /var/log/maid/ops-audit.jsonl.
  • Escalation: see ./escalation-contacts.md.template.

Symptoms

Summary

  • AI dialogue queue depth grows; NPC responses get slow or empty.
  • maid-admin status --instance <inst> shows redis: unreachable.
  • Engine journal: redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
  • /readyz is typically still green — Redis is cache-only in MAID v1 and is not on the critical path for core gameplay (login, movement, combat, persistence).

Detection

  • maid_redis_connection_errors_total rate > 0 for > 60 s.
  • maid_ai_dialogue_queue_depth climbing.
  • Doctor: maid-admin doctor --phase runtime --instance <inst> reports redis_reachable: warn (warn, not fail — see "Cache-only" note below).

Blast radius

  • Players affected: anyone interacting with AI NPCs sees degraded dialogue (longer latency, no recall of recent in-conversation context).
  • Core gameplay (look / move / get / drop / combat / chat between players) is unaffected.
  • Data at risk: none persistent. Redis in MAID v1 is a cache. NPC episodic memory lives in Postgres via the consolidation pipeline; Redis only holds the working-set cache and dialogue session state. When Redis returns, the cache repopulates from PG on demand.
  • AI/external systems: prompt-build context that depends on cached memory / relationship lookups will refetch from PG; expect a brief latency spike during warm-up.

Cache-only invariant: if a future change makes Redis authoritative (e.g., moves the leader-lease store off Postgres into Redis), this runbook MUST be revised. As of M1.2 the lease store is Postgres (leader_lock table); Redis is not the lease store. N/A: "Redis hosts the leader lease" — false for MAID v1.

Diagnostic Steps

First 5 minutes (LITERAL commands)

# 1. Confirm scope: gameplay should still work. Do NOT declare maintenance unless players report login failures.
maid-admin status --instance <inst> --json | jq '.redis, .ai_dialogue'

# 2. Capture state
sudo journalctl -u redis --since "-15 min" \
  > ./incident-redis-$(date -u +%Y%m%dT%H%M%SZ).log
sudo systemctl status redis --no-pager

# 3. Is Redis even running?
redis-cli ping || echo "redis ping failed exit=$?"

# 4. Disk OK on /var/lib/redis ? (Redis refuses writes when AOF/RDB cannot persist)
df -h /var/lib/redis

Investigation

Decision tree:

systemctl status redis
├── active (running)?
│     └── yes → connectivity / auth issue
│         - check bind address vs MAID_REDIS_URL
│         - check `requirepass` vs the URL's password
│         - `redis-cli config get maxmemory` — full?
└── not running?
      └── journalctl -u redis --since "-30 min"
          - "Can't save in background: fork: Cannot allocate memory" → host OOM, see ./oom_loop.md
          - "Write error saving DB on disk: No space left" → see ./disk_full.md
          - "MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist on disk" → check disk
          - clean exit / `systemctl stop` → someone stopped it; check audit and `last`

Useful queries once Redis is back: 

redis-cli info memory | grep -E 'used_memory_human|maxmemory_human|maxmemory_policy'
redis-cli info persistence | grep -E 'loading|rdb_last_save_time|aof_enabled'
redis-cli info clients
redis-cli --bigkeys                  # find any pathological keys causing memory bloat

Resolution Steps

Mitigation

  • Redis crashed cleanly: 

    sudo systemctl restart redis
sleep 2
redis-cli ping
    Engine reconnects automatically. Warn AI users: dialogue context for in-flight conversations is lost (working set was in Redis). NPC long-term memory is intact (lives in Postgres).

  • Redis disk-full / persistence failures:

  • See ./disk_full.md. Free space on /var/lib/redis.

  • As an emergency, disable AOF and switch to RDB-only: 

    redis-cli config set appendonly no
    File a follow-up ticket to re-enable AOF once disk is healthy.

  • Redis evicting hot keys (maxmemory_policy=allkeys-lru with high eviction rate):

  • Cache is undersized for current player population. Either bump maxmemory in /etc/redis/redis.conf and restart, or shed AI load: 

    # `maid ops ai disable` is an M9 stub; use the functional kill-switch.
sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops kill-switch trip "redis cache saturated" --confirm --instance <inst>
    See ./ai_provider_outage.md for the AI-disable / player-comms recipe.

  • Redis corruption (rare; AOF/RDB load fails): 

    sudo systemctl stop redis
sudo mv /var/lib/redis/dump.rdb /var/lib/redis/dump.rdb.corrupt.$(date -u +%s)
sudo mv /var/lib/redis/appendonly.aof /var/lib/redis/appendonly.aof.corrupt.$(date -u +%s) 2>/dev/null || true
sudo systemctl start redis
    This discards the cache. Acceptable because Redis is cache-only. Engine repopulates from PG on next access.

Recovery

  • redis-cli ping returns PONG.
  • maid-admin doctor --phase runtime --instance <inst> redis_reachable: pass.
  • maid_ai_dialogue_queue_depth returns to baseline within ~5 min.
  • If AI was disabled: maid ops kill-switch reset --instance <inst>.
  • No player comms required if /readyz stayed green throughout.

Post-incident

  • File ticket with: Redis version, crash class, time to detect, time to recover.
  • Consider raising maxmemory if eviction was the trigger.
  • If Redis went down for > 1 h during peak, evaluate whether NPC dialogue UX needs a graceful "NPC seems distracted..." fallback message instead of the current empty-response timeout.

Escalation