Runbook: OOM loop / engine restart cycle¶

Symptom¶

systemctl status maid-engine@<inst>.service shows recent restarts (Restart=on-failure cycling).
journalctl -u maid-engine@<inst> repeats: systemd[1]: maid-engine@<inst>.service: Main process exited, code=killed, status=9/KILL or Failed with result 'oom-kill'.
dmesg | grep -i 'killed process' shows the engine PID killed by OOM.
/readyz flaps green/red as the engine restarts.
Players see repeated disconnects every 30-90 s.

Detection¶

maid_engine_restart_count_total increases > 2 in 5 min.
systemd Restart=on-failure triggering > RestartLimit (engine unit enters failed state).
Watchdog metric: maid_watchdog_unclean_exit_total > 0 in the last 10 min (see M1.2-B watchdog).
No built-in engine webhook fires a named engine_oom_loop event. The observability webhook bridge (MAID_OBSERVABILITY_WEBHOOK_*) only POSTs periodic metric snapshots; wire an external Prometheus/Alertmanager rule on maid_engine_restart_count_total / maid_watchdog_unclean_exit_total to page, and notify players manually with maid ops announce.

Blast radius¶

Players affected: ALL on this instance. Each restart drops every session.
Data at risk: anything in the persistence dirty queue between two restarts; the watchdog flushes on SIGTERM but bypasses cleanup on os._exit from the wedged-loop path (see plan R10.B).
AI/external systems: dialogue sessions are torn down on each restart; Redis-cached working set survives but PG-side conversation logs may show truncated turns.

Prerequisites¶

Interim maid CLI (MAID_SRC): there is no system-wide maid binary until M1.4 (install.sh leaves /opt/maid/venv empty), so the CLI runs from a source checkout. export MAID_SRC=/path/to/maid-checkout and build its venv once: sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv: /opt/maid/current/.uv/bin/uv). Commands below run "${MAID_SRC}/.venv/bin/maid" as the maid-engine user; inside bash -c blocks MAID_SRC is forwarded via --preserve-env=MAID_SRC.
Tools: maid-admin, systemctl, journalctl, dmesg, py-spy (R9.A.12 — installed via install.sh), ps, cat /proc/<PID>/status.
Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units and sudo -u maid-engine … service-user shells, none of which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) grants (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root).
Env file: /etc/maid/<inst>.env.
Source checkout (MAID_SRC): scripts/install.sh is repo-only and is not installed at /opt/maid/current/bin/ (staging is deferred to M1.4). export MAID_SRC=/path/to/maid-checkout before the rollback step; it runs sudo bash "${MAID_SRC}/scripts/install.sh" --rollback-version --yes.
Paths to know:
Unit drop-in: /etc/systemd/system/maid-engine@.service.d/
Engine PID: cat /run/maid-engine/<inst>/engine.pid (when alive)
Heap dump output: ./engine-heap-<ts>.txt
Audit access: tail -f /var/log/maid/ops-audit.jsonl.
Escalation: see ./escalation-contacts.md.template.

First 5 minutes (LITERAL commands)¶

# 1. Shed AI load first — it is the most common runaway memory source
# `maid ops ai disable` is an M9 stub; use the functional kill-switch.
sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops kill-switch trip "oom investigation" --confirm --instance <inst>

# 2. Flush the persistence dirty queue so the next restart loses less.
#    `maid ops flush` is an M9 stub — force a synchronous save from an
#    in-game admin session with `@persistence flush` (IMPLEMENTOR) or `@save`.
#    (A clean `systemctl stop`/`restart` also flushes pending saves on SIGTERM.)

# 3. Capture state BEFORE deciding kill-vs-let-restart
sudo journalctl -u maid-engine@<inst>.service -n 500 \
  > ./incident-engine-$(date -u +%Y%m%dT%H%M%SZ).log
sudo dmesg -T | grep -i -A2 'oom\|killed process' \
  > ./incident-dmesg-$(date -u +%Y%m%dT%H%M%SZ).log
sudo systemctl status maid-engine@<inst>.service --no-pager > /dev/null  # just to refresh

# 4. Check MemoryMax headroom in the unit
sudo systemctl show maid-engine@<inst>.service \
  --property=MemoryMax,MemoryHigh,MemoryCurrent,TasksMax,TasksCurrent

Investigation¶

Step 1: is it the engine, or the host, that ran out of memory?¶

# Host memory pressure right now
free -h
cat /proc/meminfo | head -5

# Last 10 min of /proc/pressure/memory (Linux PSI)
cat /proc/pressure/memory

If host free shows plenty of RAM but the engine still got killed, the kill is from the cgroup MemoryMax limit — bump the limit (see Mitigation) or fix the leak.
If host is exhausted, another tenant on the box (Postgres? a backup job?) is the real culprit. Check:
```
ps aux --sort=-rss | head -15
systemd-cgtop -n 1 -m
```

Step 2: capture a heap dump (while engine is alive between restarts)¶

The engine restart cycle gives you a short window. Catch it on the next start:

# Watch the unit and grab the PID the moment it appears
while :; do
  PID=$(systemctl show -p MainPID --value maid-engine@<inst>.service)
  if [ "$PID" != "0" ] && kill -0 "$PID" 2>/dev/null; then
    echo "engine PID=$PID; dumping"
    sudo py-spy dump --pid "$PID" \
      > ./engine-heap-$(date -u +%Y%m%dT%H%M%SZ).txt 2>&1 || true
    sudo cat /proc/"$PID"/status \
      | grep -E 'Vm|Threads|FDSize' \
      > ./engine-procstatus-$(date -u +%Y%m%dT%H%M%SZ).txt
    break
  fi
  sleep 0.5
done

If py-spy is not on the host, install via the procedure documented in ./RB22_python_uv.md — it should be pre-installed by install.sh per R9.A.12; absence is itself an incident worth noting.

Step 3: decide kill-vs-let-systemd-retry¶

If the engine is wedged but NOT consuming new memory (rare — usually means a hot loop without alloc): let systemd retry; the watchdog will os._exit(143) within 60 s anyway.

If memory keeps climbing on each restart (warm-cache leak, unbounded dialogue queue, etc.): stop the unit and contain damage:

sudo systemctl stop maid-engine@<inst>.service
# Notify externally via webhook. `maid ops maintenance` is an M9 stub, and
# with the engine stopped there is nothing to gate — new logins already fail.
# `maid ops announce` reads MAID_BRIDGES_WEBHOOK_URLS from its own process
# env, so source the instance env with auto-export (a bare --preserve-env
# of a never-exported var forwards nothing):
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a;
   exec "${MAID_SRC}/.venv/bin/maid" ops announce --severity warn \
     --message "engine memory leak; service down, ETA 30m"'

Step 4: identify the leaker from the heap dump¶

Grep for high-allocation stacks. Common offenders observed in MAID:
EnrichedPromptBuilder retaining conversation context past TTL.
MemoryService extraction queue without back-pressure.
GridManager holding the full wilderness in RAM (no eviction).
Match the offender against recent deploys (git log --since "-7d"). Memory loops often correlate with the previous release.

Mitigation¶

Reduce AI pressure (already done in step 1): keep AI disabled until the leak is fixed.

Bump cgroup MemoryMax temporarily, only if the host has headroom:

sudo mkdir -p /etc/systemd/system/maid-engine@.service.d
sudo tee /etc/systemd/system/maid-engine@.service.d/95-resource-override.conf <<EOF
[Service]
MemoryMax=4G
MemoryHigh=3.5G
EOF
sudo systemctl daemon-reload
sudo systemctl restart maid-engine@<inst>.service

See ./RB19_resource_exhaustion.md for the full resource-cap override procedure.

Roll back to the last known good release if the leak coincided with a recent deploy:
```
sudo bash "${MAID_SRC}/scripts/install.sh" --rollback-version --yes
sudo systemctl restart maid-engine@<inst>.service
```
Note: --rollback-version is a BOOLEAN flag; it reads the prior release from /opt/maid/versions/PREVIOUS (scripts/install.sh:1341-1370) and requires --yes to swap the current symlink. Do NOT use bare --rollback — that is the per-instance UNINSTALL flag (destructive). See ./rollback.md.

Recovery¶

sudo systemctl status maid-engine@<inst>.service → active (running) with no restart events for ≥ 10 min.
maid-admin status --instance <inst> --json | jq .uptime_s ≥ 600.

MemoryCurrent plateaus instead of climbing:

watch -n 5 'systemctl show maid-engine@<inst>.service \
  --property=MemoryCurrent,TasksCurrent'

maid-admin doctor --phase runtime --instance <inst> all green.
Re-enable AI if root cause was not AI-side (maid ops ai enable is an M9 stub — use the functional kill-switch): maid ops kill-switch reset --instance <inst>.
Announce the all-clear to players via the admin broadcast (maid ops maintenance off is an M9 stub — no login gate to lift).

Post-incident¶

File ticket with: heap dump, dmesg excerpt, restart timeline, suspect module, was it correlated with a deploy.
Add a regression test or memory bound to the offending code path.
If MemoryMax was bumped as a workaround, file follow-up to revert the override once the leak is fixed.

Escalation¶

Solo path: shed AI → flush → rollback if recent deploy → restart.
Hosting console URL: see ./escalation-contacts.md.template.
DNS registrar URL: see ./escalation-contacts.md.template.
Comms channel URL: see ./escalation-contacts.md.template.
Peer operator: see ./escalation-contacts.md.template.

If you cannot recover within 30 minutes:

# `maid ops maintenance` is an M9 stub. Notify players externally via
# webhook, then do a clean stop. Source the instance env with auto-export
# so announce inherits MAID_BRIDGES_WEBHOOK_URLS (a bare --preserve-env of
# a never-exported var forwards nothing):
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a;
   exec "${MAID_SRC}/.venv/bin/maid" ops announce --severity warn \
     --message "engine OOM; taking the service down, ETA <ETA>"'

and consider a clean stop + investigate next day rather than fighting the restart loop in production.

Alternate alerting path (no Prometheus)¶

Operators without a Prometheus / Alertmanager stack get the same coverage via the maid-doctor timer + alert-script pair shipped in PR-C. See deploy/monitoring/doctor-alert.sh and packaging/systemd/maid-doctor@.timer.

Enable per instance:

sudo systemctl enable --now maid-doctor@<inst>.timer

On every 5-minute tick the timer runs scripts/maid-doctor.sh --phase runtime --json, pipes the result into doctor-alert.sh, which then:

Writes /var/lib/node_exporter/textfile_collector/maid_doctor.prom (metrics maid_doctor_last_run, maid_doctor_last_status, maid_doctor_failed_checks_total) so any future scraper picks up the most recent doctor verdict without re-running it.
On warn or fail, emails the on-call address read from /etc/maid-engine/<inst>/oncall.env (ONCALL_EMAIL).

This is the default monitoring path for solo home-lab installs; Prometheus is the optional add-on for shops that want graphs/history.