Skip to content

Runbook: RB19 — Resource exhaustion (TasksMax / MemoryMax / fd)

Symptom

  • Engine fails to spawn threads / coroutines / subprocesses; journal: OSError: [Errno 11] Resource temporarily unavailable or RuntimeError: can't start new thread.
  • Engine fails to open files / sockets; journal: OSError: [Errno 24] Too many open files.
  • systemd journal: service: Killing process N (...) with signal SIGKILL attributed to TasksMax or MemoryMax.
  • /readyz flaps or fails because the engine cannot accept new connections.
  • See ./oom_loop.md if the symptom is repeated restarts due to MemoryMax. This runbook is for in-place exhaustion diagnosis and unit-level cap adjustments.

Detection

  • maid_engine_open_fds near the unit's LimitNOFILE.
  • maid_engine_thread_count near TasksMax.
  • maid_engine_rss_bytes approaching MemoryHigh / MemoryMax.
  • systemd journal: Reached MemoryMax limit or tasks-max reached.
  • Doctor: maid-admin doctor --phase runtime --instance <inst> reports resource_headroom: warn or fail.

Blast radius

  • Players affected: ALL on this instance — new sessions fail; existing sessions may hang on resource-bound operations.
  • Data at risk: in-flight writes if the engine is killed by systemd before draining.
  • AI/external systems: AI dialogue spawns extra coroutines; AI load amplifies thread/fd usage.

Prerequisites

  • Interim maid CLI (MAID_SRC): there is no system-wide maid binary until M1.4 (install.sh leaves /opt/maid/venv empty), so the CLI runs from a source checkout. export MAID_SRC=/path/to/maid-checkout and build its venv once: sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv: /opt/maid/current/.uv/bin/uv). Commands below run "${MAID_SRC}/.venv/bin/maid" as the maid-engine user; inside bash -c blocks MAID_SRC is forwarded via --preserve-env=MAID_SRC.
  • Tools: maid-admin, systemctl, journalctl, cat /proc/<PID>/status, ls /proc/<PID>/fd, ss -tnp.
  • Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units, which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) does not grant (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root).
  • Env file: /etc/maid/<inst>.env.
  • Paths to know:
  • Unit drop-in dir: /etc/systemd/system/maid-engine@.service.d/
  • Engine PID: cat /run/maid-engine/<inst>/engine.pid
  • Audit access: tail -f /var/log/maid/ops-audit.jsonl.
  • Escalation: see ./escalation-contacts.md.template.

First 5 minutes (LITERAL commands)

# 1. Find the engine PID and current resource counters at the cgroup level
sudo systemctl status maid-engine@<inst>.service --no-pager
sudo systemctl show maid-engine@<inst>.service \
  --property=MainPID,MemoryCurrent,MemoryHigh,MemoryMax,TasksCurrent,TasksMax,IOPressure,CPUUsageNSec

# 2. Per-process resource view
PID=$(systemctl show -p MainPID --value maid-engine@<inst>.service)
echo "engine PID=$PID"
sudo cat /proc/"$PID"/status \
  | grep -E 'Vm(Peak|RSS|Size|HWM)|Threads|FDSize'

# 3. Open file descriptors right now
sudo ls /proc/"$PID"/fd | wc -l
sudo cat /proc/"$PID"/limits | grep -E 'open files|processes'

# 4. Current connection load
sudo ss -tnp | grep -c "pid=$PID"

# 5. Capture state for post-incident
sudo journalctl -u maid-engine@<inst>.service -n 300 \
  > ./incident-resource-$(date -u +%Y%m%dT%H%M%SZ).log

Investigation

Step 1: which resource is exhausted?

Compare TasksCurrent vs TasksMax, MemoryCurrent vs MemoryMax, and ls /proc/<PID>/fd | wc -l vs LimitNOFILE from /proc/<PID>/limits. Whichever is closest to its ceiling is the cap you need to bump or whose leak you need to fix.

Step 2: thread / task exhaustion

# How many threads does the engine have? Where are they?
sudo cat /proc/"$PID"/status | grep '^Threads'
# Per-thread (top-level), see what they're doing
sudo ls /proc/"$PID"/task | wc -l
sudo cat /proc/"$PID"/task/*/comm 2>/dev/null | sort | uniq -c | sort -rn | head
A pathological thread count (thousands) usually points at: - Unbounded asyncio.create_task in a hot path → file ticket. - Subprocess spawning runaway (rare in MAID; check off-tick LLM queue). - ThreadPool not capped.

Step 3: fd exhaustion

# What KIND of fd is dominating?
sudo ls -l /proc/"$PID"/fd \
  | awk '{print $11}' | sed 's|/[0-9]\+$||' | sort | uniq -c | sort -rn | head
Common culprits in MAID: - socket: count high → connection leak (sessions not being closed) or AI provider HTTP client leak. Check ss -tnp for connections to AI provider endpoints in CLOSE_WAIT. - anon_inode:[eventpoll] high → epoll leak (asyncio loops not being cleaned up). - regular files high → check the engine's log/file handle usage.

Step 4: memory exhaustion

See ./oom_loop.md for the heap-dump procedure (py-spy dump). For non-OOM-killed memory pressure (cgroup MemoryHigh throttling without an outright kill), the same procedure applies; capture a snapshot first, then decide bump-vs-fix.

Mitigation

Immediate: shed load

# Trip the AI kill switch (functional; largest single source of bursty
# threads/fds). `maid ops ai disable` is an M9 stub — use kill-switch.
sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops kill-switch trip "resource exhaustion" --confirm --instance <inst>

# `maid ops drain` (graceful no-new-logins drain) is an M9 stub with no
# functional equivalent yet. Tripping the AI kill switch above sheds the
# dominant thread/fd load; if that is insufficient, stop the engine cleanly
# with `sudo systemctl stop maid-engine@<inst>.service` (final save on SIGTERM).

Bump the caps (override drop-in)

Create a drop-in so the override survives package upgrades: 

sudo mkdir -p /etc/systemd/system/maid-engine@.service.d

sudo tee /etc/systemd/system/maid-engine@.service.d/95-resource-override.conf <<'EOF'
[Service]
# Memory
MemoryMax=4G
MemoryHigh=3.5G

# Tasks (threads + processes)
TasksMax=4096

# File descriptors
LimitNOFILE=65535
EOF

# Apply
sudo systemctl daemon-reload
sudo systemctl restart maid-engine@<inst>.service

# Verify the new caps are in effect
sudo systemctl show maid-engine@<inst>.service \
  --property=MemoryMax,MemoryHigh,TasksMax,LimitNOFILE

Bumping caps is mitigation, not fix. If usage was 95 % of the old cap, it will be 95 % of the new cap soon. File a follow-up.

If the engine is wedged and unresponsive

  • Capture state first (/proc/<PID>/status, journal).
  • Restart the unit; systemd takes care of teardown: 
    sudo systemctl restart maid-engine@<inst>.service

Recovery

  • MemoryCurrent / TasksCurrent plateau below their respective caps for ≥ 10 min: 
    watch -n 5 'systemctl show maid-engine@<inst>.service \
  --property=MemoryCurrent,TasksCurrent,MemoryMax,TasksMax'
  • ls /proc/<PID>/fd | wc -l stable.
  • maid-admin doctor --phase runtime --instance <inst> resource_headroom: pass.
  • Re-enable AI if cause was not AI-side: maid ops kill-switch reset --instance <inst>.
  • No MAID_* alert active.

Post-incident

  • File ticket with: which resource ran out, current vs cap at the moment of incident, suspect leak source, did bumping caps fix it or did a real leak need fixing.
  • If override drop-in was added, file follow-up to revert it once the leak is fixed.
  • Add a metric & alert for the resource that exhausted if one is missing.
  • Update this runbook if a new resource class showed up.

Escalation