Runbook: Persistence save queue growing¶

Symptom¶

maid_persistence_save_queue_oldest_age_seconds exceeds 2× the configured MAID_PERSISTENCE_SAVE_INTERVAL (default 300 s → alert at 600 s) and keeps climbing.
Engine journal: occasional warnings save batch took N.NNs (slow).
Players notice that loot drops / inventory changes take a noticeable beat to commit.
Doctor: maid-admin doctor --phase runtime --instance <inst> reports persistence_lag: warn or fail.

Detection¶

maid_persistence_save_queue_oldest_age_seconds > 2 * MAID_PERSISTENCE_SAVE_INTERVAL for > 5 min.
maid_persistence_save_queue_depth monotonic-growing for > 5 min.
No built-in engine webhook fires a named persistence_lag event. The observability webhook bridge (MAID_OBSERVABILITY_WEBHOOK_*) only POSTs periodic metric snapshots; wire an external Prometheus/Alertmanager rule on maid_persistence_save_queue_oldest_age_seconds to page, and notify players manually with maid ops announce.

Blast radius¶

Players affected: ALL on this instance; risk increases the longer the queue grows.
Data at risk: any entity mutation queued after the oldest unsaved-at timestamp. On crash, that window is lost; the engine restarts from the last successful save.
AI/external systems: dialogue and memory writes also queue here; same window applies.

Prerequisites¶

Tools: maid-admin, jq, psql, journalctl, systemctl.
Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units and sudo -u postgres … shells, none of which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) grants (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root).
Env file: /etc/maid/<inst>.env — MAID_PERSISTENCE_SAVE_INTERVAL, MAID_PERSISTENCE_BATCH_SIZE.
Source checkout (MAID_SRC): scripts/install.sh is repo-only and is not installed at /opt/maid/current/bin/ (staging is deferred to M1.4). export MAID_SRC=/path/to/maid-checkout before the rollback step; it runs sudo bash "${MAID_SRC}/scripts/install.sh" --rollback-version --yes.
Paths to know:
/var/lib/maid-engine/<inst>/ (dirty-tracker on-disk WAL, if enabled)
Engine pidfile: /run/maid-engine/<inst>/engine.pid
Audit access: tail -f /var/log/maid/ops-audit.jsonl.
Escalation: see ./escalation-contacts.md.template.

First 5 minutes (LITERAL commands)¶

# 1. Snapshot the queue state
maid-admin status --instance <inst> --json | jq '.persistence'
# Expect fields: queue_depth, oldest_age_s, last_save_at,
#                last_save_duration_s, batches_in_flight

# 2. Capture engine log
sudo journalctl -u maid-engine@<inst>.service -n 300 \
  > ./incident-engine-$(date -u +%Y%m%dT%H%M%SZ).log

# 3. Quick mitigation: drain the queue NOW. `maid ops flush` is an M9 stub —
#    force a synchronous save from an in-game admin session (IMPLEMENTOR):
#      @persistence flush      (drains + saves all dirty entities)
#      @save                   (triggers one immediate save cycle)
#    A clean `systemctl stop`/`restart` also flushes pending saves on SIGTERM.

# 4. Re-check
maid-admin status --instance <inst> --json | jq '.persistence'

If flush returns 0 and queue_depth is back near 0, the queue was a transient spike (long tick, GC pause, etc.). Continue to Investigation to find the cause. If flush times out or queue_depth keeps growing, treat this as DB latency / DB pressure — see Mitigation.

Investigation¶

Step 1: is DB latency the bottleneck?¶

# Engine-side timing of last N save batches
maid-admin status --instance <inst> --json \
  | jq '.persistence.recent_batches // empty'

# PG-side: any locks or long txns?
sudo -u postgres psql -d maid_<inst> -c "
  select pid, now()-xact_start as txn_age, state, wait_event_type, query
  from pg_stat_activity
  where xact_start is not null
  order by xact_start;"

# Disk write latency on the DB volume
iostat -xm 1 5    # if sysstat is installed
# or
cat /sys/block/sd?/stat   # raw counters

- If PG shows long txns or wait_event_type=Lock, DB is the bottleneck. Go to ./db_down.md — even if PG is up, the same decision tree applies (slow vs unreachable). - If iostat shows %util > 90 sustained, the disk is saturated. May also be a sign of ./disk_full.md imminent — check free space.

Step 2: is the engine the bottleneck?¶

# Are tick durations also slow?
maid-admin status --instance <inst> --json \
  | jq '{tick_p50_ms: .tick.p50_ms, tick_p99_ms: .tick.p99_ms,
          tick_lag_s: .tick.lag_s}'

# Recent slow saves attributable to a specific system?
sudo journalctl -u maid-engine@<inst>.service --since "-30 min" \
  | grep -E 'slow_save|serialize_ms|persistence.batch' | tail -50

- High serialize_ms for a particular component type → newly-deployed serializer is slow. Check recent commits:

git -C /opt/maid/current log --since "-7 days" --oneline -- 'packages/*/src/**'

If correlated with a deploy, consider rollback per ./rollback.md.

Step 3: is a system flooding the dirty queue?¶

maid-admin status --instance <inst> --json \
  | jq '.persistence.dirty_by_component_type // empty'

A pathological emitter (e.g., a system marking the same component dirty every tick) shows here. Identify and either: - temporarily disable the offending system, or - restart the engine after rolling back to last known good.

Mitigation¶

Transient spike: the flush in First 5 minutes is enough. Watch the metric for 10 min to confirm it stays drained.
DB latency the cause: follow ./db_down.md. The save queue is a symptom, not the root cause.
Disk pressure the cause: follow ./disk_full.md.

Slow serializer / new deploy regression:

sudo bash "${MAID_SRC}/scripts/install.sh" --rollback-version --yes
sudo systemctl restart maid-engine@<inst>.service
# No separate flush needed: the restart's SIGTERM stop performs a final
# save (`maid ops flush` is an M9 stub anyway). Confirm the queue drained:
maid-admin status --instance <inst> --json | jq '.persistence'

--rollback-version is a BOOLEAN flag; it reads the prior release from /opt/maid/versions/PREVIOUS (scripts/install.sh:1341-1370) and requires --yes to swap the current symlink. Do NOT use bare --rollback — that is the per-instance UNINSTALL flag (destructive). See ./rollback.md.

Runaway dirty-queue producer (a system marking too much dirty):
As a last resort, increase MAID_PERSISTENCE_BATCH_SIZE and decrease MAID_PERSISTENCE_SAVE_INTERVAL in /etc/maid/<inst>.env and restart. This only helps if the DB can absorb a larger batch.
File a ticket against the offending system for proper back-pressure.

Recovery¶

maid_persistence_save_queue_oldest_age_seconds < MAID_PERSISTENCE_SAVE_INTERVAL sustained for ≥ 10 min.
maid_persistence_save_queue_depth returns to baseline (single digits for an idle world, tens-to-low-hundreds for active play).
maid-admin doctor --phase runtime --instance <inst> persistence_lag: pass.
No MAID_* alert active.
No player comms required if /readyz stayed green throughout.

Post-incident¶

File ticket with: queue metric chart, suspected component, was a deploy involved, time to detect, time to drain.
If you bumped batch-size/save-interval as a workaround, file a follow-up to revert once the root cause is fixed.
Update this runbook with any new signal you used to diagnose.

Escalation¶

Solo path: flush → identify bottleneck (DB / disk / serializer) → fix or rollback.
Hosting console URL: see ./escalation-contacts.md.template.
DNS registrar URL: see ./escalation-contacts.md.template.
Comms channel URL: see ./escalation-contacts.md.template.
Peer operator: see ./escalation-contacts.md.template.

If queue keeps growing despite mitigation, the engine WILL eventually OOM as the dirty set inflates. Pre-empt that:

# `maid ops maintenance`/`maid ops flush` are M9 stubs. Notify players via
# the functional in-game broadcast, force a save from an in-game admin
# session (`@persistence flush`, IMPLEMENTOR), then do a clean restart
# (SIGTERM performs a final save):
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"persistence backlog, draining, ETA <ETA>"}'
sudo systemctl restart maid-engine@<inst>.service

Alternate alerting path (no Prometheus)¶

Operators without a Prometheus / Alertmanager stack get the same coverage via the maid-doctor timer + alert-script pair shipped in PR-C. See deploy/monitoring/doctor-alert.sh and packaging/systemd/maid-doctor@.timer.

Enable per instance:

sudo systemctl enable --now maid-doctor@<inst>.timer

On every 5-minute tick the timer runs scripts/maid-doctor.sh --phase runtime --json, pipes the result into doctor-alert.sh, which then:

Writes /var/lib/node_exporter/textfile_collector/maid_doctor.prom (metrics maid_doctor_last_run, maid_doctor_last_status, maid_doctor_failed_checks_total) so any future scraper picks up the most recent doctor verdict without re-running it.
On warn or fail, emails the on-call address read from /etc/maid-engine/<inst>/oncall.env (ONCALL_EMAIL).

This is the default monitoring path for solo home-lab installs; Prometheus is the optional add-on for shops that want graphs/history.