Skip to content

Operator Handbook

Your first day on the job. Read this once, end-to-end. After that it is a quick-reference: when something looks wrong you should know which runbook to open, which metric to look at, and which CLI verb to type — without re-reading.

This document does NOT duplicate runbook content. Every fix procedure lives in docs/runbooks/. This handbook is the index.

You inherit a deployment that satisfies:

  • The 14 steps in bare_metal.md have all run to completion on this host.
  • maid-doctor.sh --phase install exits 0.
  • The security checklist has been walked top-to-bottom, the must-print runbooks are in the rack, and the escalation-contacts file is filled in.

If any of the above is not true, stop here and finish bare-metal install first.

Mental model

MAID is a single-process Python service that wakes up at a fixed tick rate (default 4Hz), advances world state, persists deltas, serves player IO, and sleeps until the next tick. There is exactly one engine process per instance, and that uniqueness is enforced at three layers, in order of trust:

  1. Filesystem PID lock. systemd's PIDFile= and the engine's /run/maid-engine/<inst>/engine.pid keep two systemctl starts from racing on the same host.
  2. Postgres advisory lock (the "leader lock"). On startup the engine acquires a session-scoped advisory lock against leader_lock (migration 0001). If a different host (or a recovered-but-stale PID) already holds it, startup fails closed with LeaderLockHeld(holder_pid=...). See runbooks/two_instances_detected.md.
  3. Tripswitch. A SECURITY DEFINER row in tripswitch (migration 0004) flipped by the engine itself when an invariant fails (disk pressure, preflight drift, lease loss without recovery). Once tripped the engine refuses all writes until an operator — not the engine — runs reset_tripswitch() via the maid_operator role. The engine cannot un-trip itself.

The tick loop dispatches systems in a fixed order. Critical durability — inventory transfers, currency moves, quest completions, builder commits — flushes synchronously per the durability matrix. Everything else is batched by the save scheduler (drain-then-re-mark; see packages/maid-engine/src/maid_engine/persistence/scheduler.py) at the configured save interval. AI calls run off-tick on a bounded queue — they cannot block the tick loop, and a circuit breaker opens fail-closed if the provider goes sideways (runbooks/ai_provider_outage.md).

A watchdog sibling unit (maid-watchdog@<inst>.service) tails the engine's heartbeat file in /run/maid-engine/<inst>/. If the heartbeat goes stale, the watchdog escalates SIGTERM -> SIGKILL and drops a one-shot marker the engine reads on next start; this surfaces as an OPS_WATCHDOG_FORCE_KILL audit row.

Deeper reading:

First-time setup

You should NOT be reading this section if the deployment is already live. If you are setting up a fresh host, stop here and follow bare_metal.md end-to-end. Come back when you see [READY] in the journal.

Common tasks

Every task below either links to a runbook (operator-time procedures) or names an ops CLI verb (one-off operations). The CLI verbs that return EX_CONFIG (78) today are still STUBS pending M9 wiring; the documented signature is stable and your wrappers can be written against it now.

Task How
Check liveness curl -sf http://localhost:9090/healthz returns 200; maid-doctor.sh --phase runtime --instance <inst> exits 0.
Check readiness curl -sf http://localhost:9090/readyz returns 200 once the leader lock is held and the DB connection pool is warm.
Deploy a hot patch (content) docs/deployment/upgrade.md § Hot reload; preferred for content/template tweaks.
Deploy engine code (no schema) docs/deployment/upgrade.md § Restart; install.sh → systemctl restart.
Deploy with migration runbooks/maintenance_window.md → maintenance + maid db migrate.
Rotate DB password runbooks/secret_rotation.md.
Rotate AI provider key runbooks/secret_rotation.md (same procedure, different secret).
Run a restore drill runbooks/restore.md. The functional restore path is deploy/scripts/restore.sh staging --instance <inst> --backup-id <BID> (run as root, env-sourced; after backup.sh verify). maid-admin restore is an M1.3 shim — it validates, takes the flock, audits, then returns "deferred" (M1.4); it does NOT restore. maid ops restore is a separate M9 stub (--mode staging not implemented). Do not rely on either.
Announce maintenance maid ops announce -m "Going down in 10m for upgrade" --eta 10m is functional but webhook-only — it POSTs to MAID_BRIDGES_WEBHOOK_URLS and does not reach in-game players or write a status page (--channels is metadata). To reach connected players use POST /api/v1/admin/broadcast. See player_comms.md.
Enter maintenance mode maid ops maintenance on --reason "deploy v0.9.1" (M9 stub today — the state toggle is not yet enforced; there is no working login gate).
Exit maintenance mode maid ops maintenance off --reason "deploy complete" (M9 stub today).
Drain (no new sessions) maid ops drain --instance <inst> (M9 stub — not implemented). When implemented it refuses new logins while keeping existing sessions until they idle out.
Drain-shutdown maid ops drain-shutdown --timeout 300s (M9 stub — not implemented). For a clean stop today use sudo systemctl stop maid-engine@<inst>.service (the engine flushes pending saves on SIGTERM).
Disable AI provider calls maid ops kill-switch trip "anthropic 5xx spike" --confirm (functional; maid ops ai disable is an M9 stub). The tick loop keeps running; NPC dialogue falls back to scripted lines. See runbooks/ai_provider_outage.md.
Re-enable AI maid ops kill-switch reset (functional; maid ops ai enable is an M9 stub).
Switch AI provider No working ops command — maid ops ai set-provider is an M9 stub. To change providers, edit MAID_AI_DEFAULT_PROVIDER (and the target provider's API-key var) in /etc/maid/<inst>.env and restart: sudo systemctl restart maid-engine@<inst>.service. For a temporary outage, prefer the kill-switch row above (keeps the engine up with scripted fallback) instead of a provider swap.
Trip AI kill-switch (manual) maid ops kill-switch trip "<why>" --confirm --operator "<you>" (top-level, functional; the reason is a positional argument and --confirm is required because the verb is DESTRUCTIVE). The nested maid ops ai kill-switch group is an M9/M1 stub — do not use it. All AI provider calls raise ProviderUnavailable until reset; tick loop unaffected. Audited as OPS_AI_KILL_SWITCH_TRIPPED. Increments maid_ai_kill_switch_reasons_count; maid_ai_kill_switch_tripped goes to 1.
Reset AI kill-switch maid ops kill-switch reset --operator "<you>" (top-level, functional; the nested maid ops ai kill-switch reset is a stub). Clears the tripped state.
Inspect AI kill-switch maid ops kill-switch status (top-level, functional; the nested maid ops ai kill-switch status is a stub) prints trip state, reason stack, and last operator.
Configure AI tick-isolation enforcement Set MAID_AI_TICK_ISOLATION_MODE=warn (default, logs + counts) or MAID_AI_TICK_ISOLATION_MODE=reject (raises TickIsolationViolation). Watch maid_ai_tick_isolation_violations_total{mode} — any non-zero rate means AI I/O is leaking into the tick loop and must be moved to an off-tick queue.
Force a flush maid ops flush --instance <inst> (M9 stub — not implemented). To force a synchronous save today, use the in-game admin command @persistence flush (or @save), or systemctl stop (which performs a final save on SIGTERM).
Force a save (one-shot) maid ops backup is an M9 stub. backup.sh is not staged under /opt/maid/current until M1.4, so run it from a source checkout: export MAID_SRC=/opt/maid/src (or wherever you cloned the repo), then sudo -u maid-engine --preserve-env=MAID_SRC bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" full --instance <inst>' (source the instance env so backup.sh sees MAID_DEPLOY_BACKUP_REMOTE; a bare sudo does not load it; --preserve-env=MAID_SRC is required because sudo's env_reset otherwise drops it before the inner shell expands the path). To force an in-process entity save, use the in-game @save admin command.
Pause / resume the tick maid ops pause-tick / maid ops resume-tick (M9 stubs — not implemented). There is no supported tick-pause today; for invasive repair, stop the engine (systemctl stop).
Reload config maid ops reload-config is an M9 stub. There is no in-place config reload. /etc/maid/<inst>.env is read only at process start, and systemctl reload sends an unhandled SIGHUP that terminates the engine (systemd then restarts it under Restart=on-failure). To apply an env change, use sudo systemctl restart maid-engine@<inst>.service and treat it as a restart (≤ 60s disconnect).
Trip the breaker (manual) Via SQL as the maid_operator role: SELECT trip_tripswitch('manual: <reason>', '{}'::jsonb, NULL, '<your-name>'). Refuses all writes engine-wide.
Reset the breaker Via SQL as maid_operator: SELECT reset_tripswitch('<your-name>', '{}'::jsonb). Only operators — not the engine — can clear it.
Erase a player (GDPR Art 17) maid ops delete-player-data --account-id <id> --confirm --ticket <DPO-ticket> (Task 5; functional today). See runbooks/player_data_erasure.md.
Export a player (GDPR Art 15) maid ops export-player-data <account_id> (deferred; documented in privacy.md).
Rollback a release sudo bash scripts/install.sh <inst> --rollback-version --yes. See runbooks/rollback.md.
Decommission an instance sudo bash scripts/install.sh <inst> --rollback --yes (preserves state on disk and DB).
Print operator status maid ops status --instance <inst> (M9 stub today); offline equivalent: maid-admin status --instance <inst> (reports offline + runbook RB22 when the engine UDS is unreachable).

Admin extension route host

Pack-owned admin HTTP routes run in an isolated child host mounted once at /admin/ext; lifecycle changes never mutate the main FastAPI route list. Pack Inventory reports the host status, current generation, endpoint count, active requests, retired generations, and reason. The authenticated GET /admin/ext/openapi.json reports the same current generation in the X-MAID-Extension-Generation header. Child docs and Redoc are not exposed.

Normal load, reload, and unload operations build and validate a complete candidate before publication. New requests acquire exactly one generation; old in-flight requests drain on their original generation. A failed candidate leaves the prior host and schema active. If the final swap fails after runtime work, MAID rolls runtime state back when possible and otherwise reports a failed/degraded pack rather than success.

For emergency rollback, set the following in the instance environment and restart:

MAID_ADMIN_DYNAMIC_ROUTE_HOST_ENABLED=false

This restores frozen-route semantics: startup pack routes remain guarded, but late load and code reload of route-owning packs show restart_required. Re-enable the default (true) only after correcting the route collision or candidate failure and restarting. Do not proxy or expose child paths without the normal admin authentication, CSRF, origin, rate-limit, and request-size middleware.

Reading metrics

The engine exposes Prometheus metrics on 127.0.0.1:9090/metrics. Pull from your aggregator over a private channel (the firewall in bare_metal.md Step 11 keeps :9090 loopback-only).

The five you check daily

Metric Healthy range Alarm condition Runbook
maid_engine_tick_lag_seconds (p99) < 0.5s @ 4Hz > 2× tick interval ⇒ warning; > 3× ⇒ watchdog escalates. oom_loop.md, save_queue_growing.md
maid_persistence_save_queue_oldest_age_seconds < MAID_PERSISTENCE_SAVE_INTERVAL (300s default) > 2× MAID_PERSISTENCE_SAVE_INTERVAL ⇒ data at risk. save_queue_growing.md
maid_backup_last_success_timestamp_seconds < 25 hours ago > 25h ⇒ RPO breached. Two failures in a row ⇒ page. ⚠ M9-pending: not written on success under the hardened maid-backup@.service (ProtectSystem=strict); until packaged, confirm via the OPS_BACKUP_COMPLETE audit row + maid-backup-failure@ alert. backup_failed.md
maid_ai_circuit_breaker_state 0 (closed) 1 (open) sustained > 5 min ⇒ provider outage. ai_provider_outage.md
maid_leader_lock_held 1 on the active host 0 on the active host ⇒ split-brain risk; STOP and read RB. two_instances_detected.md

How to read the dashboard rows

The shipped Grafana dashboard (deploy/monitoring/dashboards/operations.json) has six rows:

  1. Tick health. Tick lag p50/p99, tick rate. Flat lines = healthy.
  2. Persistence. Save queue depth, oldest write age, flush rate. A growing queue without a corresponding error rate means you are running below MAID_PERSISTENCE_SAVE_INTERVAL's capacity — scale down writes or up the interval.
  3. Connections. Telnet sessions, web sessions, login rate. A sustained drop usually means firewall or DNS, not engine.
  4. DB / cache. Postgres connection pool utilisation; Redis latency. Sustained > 80% pool ⇒ tune MAID_DB_POOL_SIZE.
  5. AI. Provider request rate, circuit-breaker state, queue depth. See runbooks/ai_provider_outage.md.
  6. Backup / DR. Last successful backup timestamp, on-host staging-disk free, off-host destination reachability.

A row that has gone red and you do NOT have a runbook for is a bug report — file an issue with the dashboard screenshot.

Reading logs

The engine emits three log streams. Know which is which:

Audit log locations

MAID writes audit data to two distinct streams with different writers, paths, and use cases. Operators investigating an incident typically need to consult both:

Stream Path Writer Use
Engine ring-buffer flush data/logs/audit/audit-YYYY-MM-DD.jsonl (engine cwd) engine AuditLogger (in-memory 1000-entry deque + periodic flush) live engine events; player-facing actions; admin commands routed through the engine
Privileged ops audit /var/log/maid/ops-audit.jsonl scripts/install.sh, maid-priv-helper, engine audit_sink (PR-C, in flight) rollback, restore, GDPR erasure, credential rotation, kill-switch trips — everything that runs as root or via the priv-helper, outside the engine's normal event bus

The two streams overlap at exactly one point: a privileged op performed via the engine (e.g. an admin issuing @purge from a live session) lands in the engine stream; a privileged op performed by the operator at the shell (e.g. sudo bash scripts/install.sh --rollback-version) lands in the ops stream.

The convenience recipe at the end of this section greps both.

The three streams

Stream Path What's in it Retention
systemd journal (engine + watchdog) journalctl -u maid-engine@<inst>.service Startup, shutdown, errors, INFO-level lifecycle. Goes to the journal so it rotates with the host. per journald config
Engine application log /var/log/maid-engine/<inst>/engine.log DEBUG/INFO/WARNING/ERROR application-level. Rotated by logrotate. 30 days on host
Audit logs (both streams above) see "Audit log locations" above One JSON line per privileged action. Append-only on disk. Used for post-incident review and GDPR / compliance. Operator-configured via MAID_OBSERVABILITY_AUDIT_RETENTION_DAYS (default 90 days; not yet enforced by the engine — see privacy.md § Implementation status). For GDPR Article 30 7-year requirements, deploy a SIEM (Splunk / Elastic / Datadog / rsyslog → object store) that tails both JSONL sinks and applies retention there.

Log levels

Engine logs use the standard Python logging levels. In production you should see:

  • DEBUG — only when MAID_DEBUG=1 is set (do not leave on).
  • INFO — once per minute or less when idle. Spam here is a bug.
  • WARNING — recoverable issues (retry, fallback, transient provider error). Aggregate, don't page.
  • ERROR — recoverable but not auto-recovered. Investigate within the SLO breach window for the affected metric.
  • CRITICAL — invariant breach (the tripswitch may have flipped). Page on every one.

Common error patterns

Log line snippet Meaning Runbook
LeaderLockHeld(holder_pid=...) Another process holds the advisory lock. two_instances_detected.md
MigrationFailure: 0005_gdpr_erasure.sql ... ROLLBACK A migration failed and was rolled back; engine refuses to start. failed_migration.md
TripswitchTripped(reason="disk pressure", ...) The breaker is open. The engine will refuse writes. disk_full.md → reset via SQL
asyncpg.exceptions.ConnectionDoesNotExistError DB connection pool can't reach Postgres. db_down.md
RedisConnectionError Cache unreachable; engine continues (degraded) but NPC dialogue queues stall. redis_down.md
BackupFailed(stage=upload, ...) Backup ran but did not upload off-host. RPO at risk. backup_failed.md
OOMKilled in journal Kernel OOM-killed the engine; systemd restarted it. oom_loop.md
AICircuitBreakerOpen(provider=anthropic) AI provider returning 5xx/429; calls failing fast. ai_provider_outage.md
OPS_WATCHDOG_FORCE_KILL (audit) Watchdog escalated SIGKILL on a previous run. Investigate prior stuck-tick / OOM.

How to find a user complaint in the audit log

The audit log indexes by account_id (UUID), not username. Resolve the username first:

sudo -u postgres psql -d "maid_${INST}" -tAc \
    "SELECT id FROM accounts WHERE username = '<player-username>';"

Then grep both audit streams for that ID — the engine writes per-day JSONL files and the ops sink appends to a single file:

ACCOUNT_ID="<the-uuid-from-above>"
sudo grep -h "\"${ACCOUNT_ID}\"" \
    data/logs/audit/audit-*.jsonl \
    /var/log/maid/ops-audit.jsonl \
    2>/dev/null | jq -s 'sort_by(.ts)'

The 2>/dev/null swallows "file not found" for whichever stream hasn't been written yet on a fresh install; the jq -s 'sort_by(.ts)' merges both streams in timestamp order so you see the full sequence (engine action → priv-helper escalation → ops-side outcome) in one view.

For very large audit logs, prefer zstdgrep against rotated files (if you have a logrotate / SIEM forwarder configured per the "Audit log locations" subsection above; the engine ships none by default). The schema of each line is in packages/maid-engine/src/maid_engine/logging/audit.py (AuditEntry) for the engine stream and in scripts/install.sh (look for ops_audit function) for the ops stream.

Operator privilege tiers (maid ops)

Every maid ops verb is classified into one of three tiers. The (forthcoming) UDS control plane gates each call on the caller's SO_PEERCRED identity — i.e. the kernel-attested Unix uid/gid of whoever is connected to the socket. You cannot spoof this from userspace, so the right answer to "who can run what" is "create the right Unix groups and put people in them".

Tier Default group Verbs
READ_ONLY maid-ops-ro status, doctor, kill-switch status, audit-log query, list-secrets
MUTATE maid-ops broadcast, announce, maintenance, flush, drain, reload-config, pause-tick, resume-tick, backup, kill-switch reset
DESTRUCTIVE maid-ops-destructive drain-shutdown, restore, delete-player-data, kill-switch trip, generate-secret

Three rules to remember:

  1. Tiers are hierarchical. A user in maid-ops-destructive automatically satisfies MUTATE and READ_ONLY — they don't need to also be in the lower-tier groups.
  2. DESTRUCTIVE verbs additionally require --confirm. Group membership alone is not enough; the operator must type a confirmation token. This is the seatbelt for "I meant to do that".
  3. Unknown verbs default to DESTRUCTIVE. Anything not in the table above fails closed. If you're rolling out a new internal verb, add it to packages/maid-engine/src/maid_engine/ops/auth_tiers.py and write a test, otherwise nobody can call it.

Site-specific group names

If your site doesn't use the default group names, override them via environment (typically in /etc/maid/environment so the engine and CLI agree):

MAID_OPS_GROUP_READ_ONLY=site-dashboards
MAID_OPS_GROUP_MUTATE=site-sre
MAID_OPS_GROUP_DESTRUCTIVE=site-sre-dba

Provisioning checklist

  • Read-only on-call: usermod -aG maid-ops-ro <user>
  • Day-to-day SRE: usermod -aG maid-ops <user>
  • Senior on-call / DBA: usermod -aG maid-ops-destructive <user>
  • Audit: getent group maid-ops-destructive should be a short list. If everyone is in it, the tiering achieves nothing.

Emergency contacts

The filled-in escalation roster lives at /etc/maid/escalation-contacts.md (mode 0640 root:maid-admin, not in git). The template is docs/runbooks/escalation-contacts.md.template.

If you cannot reach the file (host loss):

  1. The roster was printed and is in the rack. Read it.
  2. If the rack is also unavailable, your runbook bundle PDFs ship with an escalation-contacts.pdf that you printed alongside.
  3. If neither is available, your DR procedure has failed; this is a process bug to fix post-incident.

Glossary

Terms that appear in runbooks, error messages, and this handbook. Sorted alphabetically.

  • Audit log — MAID writes audit data to two distinct streams: (1) the engine ring-buffer flush at data/logs/audit/audit-YYYY-MM-DD.jsonl (engine AuditLogger, in-memory deque flushed periodically; covers admin commands and player-facing privileged actions routed through the engine), and (2) the privileged ops sink at /var/log/maid/ops-audit.jsonl (written by scripts/install.sh, maid-priv-helper, and the PR-C audit_sink module; covers rollback, restore, GDPR erasure, credential rotation, kill-switch trips — everything that runs as root or via the priv-helper). Both are append-only JSONL. The engine does not rotate or purge either stream — the operator configures retention via SIEM or logrotate. When investigating an incident, grep both; see "Audit log locations" above for the combined recipe. See also privacy.md § Implementation status.
  • Backup tier — backup retention layers: daily for 14 days, weekly for 8 weeks, monthly for 90 days. An erasure is not complete until the monthly tier rolls over.
  • Breaker / tripswitch — see Tripswitch.
  • Circuit breaker — distinct from tripswitch: a per-provider exponential-backoff gate around AI calls. Opens after N consecutive failures; closes after a successful probe.
  • ContentPack — a pluggable module (maid-stdlib, maid-classic-rpg, maid-tutorial-world, …) that contributes systems, components, commands, and content to the engine. See docs/concepts/content-packs.md.
  • Drain — refuse new sessions but keep existing ones; idle them out cleanly before shutdown.
  • DR — disaster recovery; the RB18_host_loss_dr.md procedure for rebuilding on a new host.
  • DPO — Data Protection Officer; the role responsible for approving GDPR Article 17 (erasure) and Article 15 (access) requests before an operator acts.
  • ECS — Entity Component System; the core data model. Entities are IDs, components are data, systems are behaviour. See docs/concepts/ecs.md.
  • EX_CONFIG (78) — the sysexits exit code reserved for "the command was understood but cannot run yet" (i.e. M9 stub).
  • /healthz, /livez, /readyz — Kubernetes-style health endpoints on :8080. /livez is "process up"; /healthz is "loop ticking"; /readyz is "leader lock held + DB pool warm + accepting traffic".
  • Heartbeat file — the file the engine touches every N seconds in /run/maid-engine/<inst>/; the watchdog reads its mtime.
  • Instance — a single named engine deployment on a host (prod, staging, tutorial). One DB, one env file, one systemd template instantiation. Regex ^[a-z][a-z0-9_]{0,30}$.
  • Leader lock — Postgres advisory lock on leader_lock that enforces "exactly one engine writer". Held for the lifetime of the engine session.
  • Lease — short-lived rights (e.g. NPC AI generation, builder edit lock) renewed by the holder until released or expired. NOT the same as the leader lock.
  • maid_operator — Postgres role granted EXECUTE on reset_tripswitch() and erase_account_cascade() (M9); the privileged operator identity.
  • maid_app__role — Postgres role the engine connects as. No DDL, no operator functions; ordinary CRUD only.
  • Off-host backup — the destination configured in MAID_DEPLOY_BACKUP_REMOTE. backup.sh does not encrypt archives (there is no backup encryption key); apply encryption-at-rest at the storage layer if you require it.
  • Off-tick queue — the bounded async work queue that AI calls live on; cannot block the tick loop.
  • Preflight — scripts/install.sh's gating checks before any state mutation. Also maid-doctor.sh --phase install.
  • Priv-helper — /usr/local/sbin/maid-priv-helper, the privileged subset of operator verbs that route through sudo per the maid-admin sudoers allowlist.
  • Save scheduler — the drain-then-re-mark async batcher that flushes dirty entities at MAID_PERSISTENCE_SAVE_INTERVAL. The "save queue" metric is this scheduler's depth.
  • Tick — one iteration of the engine loop. Default rate 4Hz (MAID_GAME_TICK_RATE=4.0).
  • Tripswitch — the singleton SQL breaker (tripswitch table + trip_tripswitch() / reset_tripswitch() functions). Operator reset only.
  • Watchdog — the standalone sibling unit (maid-watchdog@<inst>.service) that tails the heartbeat and SIGTERM/SIGKILL-escalates a stuck engine.

Where to go next

You are done with the handbook when you can answer, without re-reading, each of these:

  1. Where do you look first when /readyz returns 503? → DB. db_down.md.
  2. The save queue alert just fired. What command do you run before reading the runbook? → maid ops status to confirm the engine is still ticking; then save_queue_growing.md.
  3. The AI provider is 5xx-ing. Does the game stay up? → Yes; AI is off-tick and the breaker fails closed. Players see scripted NPC fallback lines. See ai_provider_outage.md.
  4. A migration failed mid-way and the engine refuses to start. What's the order: rollback symlink, restore DB, or both? → It depends — read the matrix in failed_migration.md.
  5. A DPO ticket asks for full erasure of alice@example.com. What's the first thing you do? → Verify the ticket against your erasure register. Never start from a player message. Then runbooks/player_data_erasure.md.

If any of those felt unfamiliar, go back and read the linked runbook once more before your next on-call shift.