Runbook: Secret Rotation¶

Use this runbook for scheduled rotation (≤ 90 days for MAID_ADMIN_SECRET_KEY per ../deployment/security_checklist.md) AND for compromise rotation (suspected leak, departing operator, lost laptop). The procedure is the same; the urgency is different — for compromise rotation, skip the maintenance-window coordination and start at Step 1 of Resolution immediately.

Prerequisites¶

Interim maid CLI (MAID_SRC): there is no system-wide maid binary until M1.4 (install.sh leaves /opt/maid/venv empty), so the CLI runs from a source checkout. export MAID_SRC=/path/to/maid-checkout and build its venv once: sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" (uv: /opt/maid/current/.uv/bin/uv). Commands below run "${MAID_SRC}/.venv/bin/maid" as the maid-engine user; inside bash -c blocks MAID_SRC is forwarded via --preserve-env=MAID_SRC.
Tools: maid-admin, psql (from postgresql-client-16), openssl rand or python -c 'import secrets; ...', ssh, a text editor.
Access: maid-admin group membership covers the priv-helper rotate-credentials verb (Cmnd_Alias MAID_ROTATE, see packaging/admin/wrapper.py:446-452). Everything else here needs root or an equivalently broad sudo grant: the DB role swap uses direct sudo -u postgres psql (the parallel-role rotation flow is not yet automated — see Limitations), the pre-rotation backup uses sudo -u maid-engine …, and unit control uses sudo systemctl/sudo journalctl — none of which the narrow maid-admin allowlist grants. There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root).
Source checkout (MAID_SRC): deploy/scripts/backup.sh is repo-only and is not staged under /opt/maid/current until M1.4, so export MAID_SRC=/path/to/maid-checkout before the backup steps below; they invoke "${MAID_SRC}/deploy/scripts/backup.sh" and forward MAID_SRC through sudo with --preserve-env=MAID_SRC.
Env file: /etc/maid/<instance>.env writable (mode 0640 root:maid-engine).
Paths to know:
/etc/maid/<instance>.env — engine env file
/etc/maid/<instance>.env.previous — last-known-good backup (you create this in Step 1 of every rotation)
/var/log/maid/ops-audit.jsonl — append-only audit log
/opt/maid/current/.uv/bin/uv — uv for building the checkout venv (uv sync)
Reference: the canonical secret inventory lives in ../deployment/security_checklist.md. Anything in the env file that is sensitive but NOT in that checklist should be added there before rotation, so the next operator finds it.
Escalation contacts: see ./escalation-contacts.md.template.
Plan a 10-minute maintenance window for scheduled rotations. Every secret stored in /etc/maid/<inst>.env is read only at engine start, so applying a rotated env value requires an engine restart — there is no in-place/SIGHUP reload (systemctl reload sends an unhandled SIGHUP that terminates the engine; systemd then restarts it under Restart=on-failure). The one exception is the admin API token, which is rotated live via the admin REST API with no restart (Section D).

Symptoms¶

Summary¶

This runbook is operator-initiated, not alert-driven. Either:

Scheduled — a calendar reminder, doctor warning that a secret is > 90 days old, or a compliance audit.
Compromise — credentials known or suspected to have leaked (force-pushed to a public repo, departing operator's laptop is gone, vendor breach, audit-log finding).

Detection¶

sudo maid-doctor.sh --phase runtime --instance <inst> warns when secret_key.last_rotated exceeds 90 days (see docs/deployment/security_checklist.md).
An operator-reported leak or compromise notification (out of band).
maid-admin doctor placeholder warning about a missing or expired API key.
DB password compromise: an unexpected entry in pg_stat_activity with application_name LIKE 'maid-engine:%' from an unknown source IP — see ./two_instances_detected.md for fencing if the suspect connection is currently active.

Blast radius¶

Scheduled rotation, env-file secret: an engine restart is required to load the new value from /etc/maid/<inst>.env (≤ ~10s reconnect). The admin API token (Section D) is the exception — it rotates live via REST with zero restart.
DB role rotation (maid_app_<inst>_v1 → _v2): one engine restart per instance. Multi-instance hosts must do each instance separately. ~10s of unavailability per instance.
MAID_ADMIN_SECRET_KEY rotation: this HS256 secret always signs the web player-client JWTs, and signs the admin-panel/REST JWTs only when MAID_ADMIN_ALGORITHM=HS256 (non-default). Under the default RS256, admin/REST tokens are signed by the RSA key pair instead (see the next bullet). There is no previous-key / grace slot, so rotating the secret and restarting immediately invalidates every web player-client session (and, under HS256, the admin/REST sessions too); those users must log in again. Telnet sessions reconnect and re-authenticate on the restart regardless.
Admin/REST RSA signing key rotation (default RS256): admin-panel and REST JWTs are signed by the RSA private key at MAID_ADMIN_PRIVATE_KEY_PATH (verified via MAID_ADMIN_PUBLIC_KEY_PATH). If those paths are unset the pair is regenerated on every start, so a restart alone already invalidates admin tokens; if pinned, replacing the PEMs + restart does. No grace slot either — admin users re-login.
AI provider key compromise: new conversations work immediately; in-flight provider calls may fail and be retried.
Backup SSH key compromise: in-flight backups MAY fail until the new key is deployed on the destination. Backup retention protects you for one cycle.
Admin API token compromise: any session using the compromised token sees an immediate 401 on the next request.

Diagnostic Steps¶

First 5 minutes (LITERAL commands)¶

# 1. Identify the instance(s) to rotate.
maid-admin instance list

# 2. Snapshot the current env file (your rollback path).
sudo cp -a /etc/maid/<inst>.env /etc/maid/<inst>.env.previous
sudo chmod 0640 /etc/maid/<inst>.env.previous
sudo chown root:maid-engine /etc/maid/<inst>.env.previous

# 3. Capture pre-rotation engine state for the audit trail.
maid-admin status --instance <inst> --json \
    > ./pre-rotation-status-$(date -u +%Y%m%dT%H%M%SZ).json
sudo journalctl -u maid-engine@<inst>.service -n 100 \
    > ./pre-rotation-journal-$(date -u +%Y%m%dT%H%M%SZ).log

# 4. Confirm doctor is green BEFORE you start (so any post-rotation
#    failure can be unambiguously attributed to the rotation).
maid-admin doctor --phase runtime --instance <inst>

Identify which secret is being rotated¶

Cross-reference the checklist in ../deployment/security_checklist.md. This runbook covers the following secret families; pick the matching section in Resolution:

Secret family	Apply mode	Player impact	Resolution section
DB app password (`PGPASSWORD` for `maid_app_<inst>_v1`)	parallel-role swap, then engine restart	~10s per instance	A
AI provider key (`MAID_AI_ANTHROPIC_API_KEY` etc.)	engine restart	brief restart (~10s)	B
Backup destination SSH key / IAM key	no engine restart needed	none if next backup OK	C
Admin API token (`/api/v1/admin/api-keys`)	live via admin REST API — no restart	old token 401 on next request	D
`MAID_ADMIN_SECRET_KEY` (HS256; player-client JWTs always, admin/REST JWTs only if `MAID_ADMIN_ALGORITHM=HS256`)	flat swap, then engine restart (single key — no grace slot)	web player-client JWTs (and, under HS256, admin/REST JWTs) invalidated at restart; users re-login	E1
Admin/REST RSA signing pair (`MAID_ADMIN_PRIVATE_KEY_PATH` / `MAID_ADMIN_PUBLIC_KEY_PATH`; default RS256)	replace PEMs (or just restart if paths unset), then engine restart (no grace slot)	admin/REST JWTs invalidated at restart; admin users re-login	E2

Verify you have a safe rollback path¶

# Env file backup exists from Diagnostic Step 2:
ls -l /etc/maid/<inst>.env.previous

# For DB rotations: confirm the OLD role still exists and works
# (you'll need to fall back to it if the new role is mis-granted).
sudo -u postgres psql -d "maid_<inst>" -c "\du maid_app_<inst>_v1"

Resolution Steps¶

Section A — DB app password (`maid_app_<inst>_v1` → `_v2`)¶

The plan-blessed pattern (plan.md R10.A.10, also packaging/postgres/README.md:38-41) is the parallel-role swap: create _v2 with the same group membership, swap the engine's PGUSER, reload the pool, then drop _v1. Grants flow through maid_app_<inst>_role so the rotation is transparent.

# 1. Generate a new strong password (record it briefly; you'll wipe
#    your scratch file after Step 4).
NEW_PG_PW="$(openssl rand -base64 32 | tr -d '/+=' | head -c 40)"
echo "${NEW_PG_PW}" > ./scratch-pgpw.txt
chmod 0600 ./scratch-pgpw.txt

# 2. Create the parallel role _v2 with the same group membership.
sudo -u postgres psql <<SQL
  CREATE ROLE maid_app_<inst>_v2 LOGIN PASSWORD '${NEW_PG_PW}';
  GRANT maid_app_<inst>_role TO maid_app_<inst>_v2;
  -- pg_signal_backend mirror (matches v1; see packaging/postgres/README.md:33)
  GRANT pg_signal_backend TO maid_app_<inst>_v2;
SQL

# 3. Update the engine env file to use _v2.
sudo sed -i.bak \
    -e "s/^PGUSER=maid_app_<inst>_v1$/PGUSER=maid_app_<inst>_v2/" \
    -e "s|^PGPASSWORD=.*$|PGPASSWORD=${NEW_PG_PW}|" \
    /etc/maid/<inst>.env
sudo chmod 0640 /etc/maid/<inst>.env
sudo chown root:maid-engine /etc/maid/<inst>.env
# (the .bak is a second rollback path next to .previous from Diag-2)

# 4. Add a pg_ident mapping for _v2 (so socket-peer auth works for
#    the unix-socket path; pattern from
#    packaging/postgres/pg_ident.snippet.template:24)
# TODO(M1.3): `maid-admin rotate-credentials` is wired into the
# wrapper at packaging/admin/wrapper.py:446-452 but the priv-helper
# implementation is not yet shipped — until then add the snippet
# manually:
sudo bash -c 'echo "maid_engine_<inst>   maid-engine   maid_app_<inst>_v2" \
    >> /etc/postgresql/16/main/pg_ident.conf'
sudo systemctl reload postgresql

# 5. Restart the engine to pick up the new credentials (env is read
#    only at process start; there is no in-place pool reload).
sudo systemctl restart maid-engine@<inst>.service

# 6. Verify the engine is connecting as _v2.
sudo -u postgres psql -d "maid_<inst>" -c "
  SELECT usename, application_name, state
    FROM pg_stat_activity
   WHERE application_name LIKE 'maid-engine:%';"
# → usename should now be maid_app_<inst>_v2 for every engine row.

# 7. Drop the old role.
sudo -u postgres psql -c "DROP ROLE maid_app_<inst>_v1;"

# 8. Wipe scratch and clean up.
shred -u ./scratch-pgpw.txt
sudo rm /etc/maid/<inst>.env.bak

Section B — AI provider key¶

# 1. Issue a new key from the provider's console (Anthropic, OpenAI,
#    etc.). Save it locally before pasting.
NEW_AI_KEY="<paste new key here>"

# 2. Update the env file. Common variables:
#      MAID_AI_ANTHROPIC_API_KEY
#      MAID_AI_OPENAI_API_KEY
#      MAID_AI_DIALOGUE_*  (see CLAUDE.md AI Dialogue section)
sudo sed -i "s|^MAID_AI_ANTHROPIC_API_KEY=.*|MAID_AI_ANTHROPIC_API_KEY=${NEW_AI_KEY}|" \
    /etc/maid/<inst>.env
sudo chmod 0640 /etc/maid/<inst>.env

# 3. Restart so the engine picks up the new key (env is read only at
#    process start — SIGHUP/`systemctl reload` would just kill it).
sudo systemctl restart maid-engine@<inst>.service

# 4. Smoke-test the new key (see Recovery below for the full check).
#    Runs as the service user with the instance env sourced so test-ai
#    reads the SAME MAID_AI_* key the engine just reloaded.
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
    'set -a; . /etc/maid/<inst>.env; set +a;
     exec "${MAID_SRC}/.venv/bin/maid" dev test-ai \
       "hello, world" --provider anthropic'

# 5. Revoke the OLD key in the provider's console. Do this AFTER
#    Step 4 succeeds, not before.

Section C — Backup destination SSH key / IAM key¶

# 0. Resolve the key path backup.sh actually uses. The transport reads
#    MAID_DEPLOY_SSH_KEY (default /var/lib/maid-engine/.ssh/id_ed25519,
#    per deploy/scripts/lib/transport-ssh.sh); it MUST stay owned by
#    maid-engine at mode 600/400 or the transport refuses it. Rotating
#    any OTHER path (e.g. a per-instance file that isn't wired into
#    MAID_DEPLOY_SSH_KEY) has no effect on backups.
KEY="$(sudo -u maid-engine bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; echo "${MAID_DEPLOY_SSH_KEY:-/var/lib/maid-engine/.ssh/id_ed25519}"')"
echo "rotating transport key: $KEY"
# If you want a dedicated per-instance key, set MAID_DEPLOY_SSH_KEY in
# /etc/maid/<inst>.env to that path FIRST (and restart nothing — backup.sh
# reads it per-invocation), then run this rotation against it.

# 1. Generate the new SSH key alongside the current one (or rotate the
#    IAM key in the cloud console). ssh-keygen writes mode 0600.
sudo -u maid-engine ssh-keygen -t ed25519 -f "${KEY}.new" -N ''

# 2. Push the new PUBLIC key ("${KEY}.new.pub") to the destination (out
#    of band, e.g. your provider's `authorized_keys` management). Verify
#    SSH works with the new key:
sudo -u maid-engine ssh -i "${KEY}.new" \
    -o StrictHostKeyChecking=accept-new \
    <backup-user>@<backup-host> \
    'echo ok'

# 3. Atomic swap. Keep the old key as .previous for rollback.
sudo -u maid-engine cp -a "${KEY}" "${KEY}.previous"
sudo -u maid-engine mv "${KEY}.new"     "${KEY}"
sudo -u maid-engine mv "${KEY}.new.pub" "${KEY}.pub"

# 4. Run one backup cycle to prove the new key works end-to-end, and
#    verify THAT backup by id (fail-closed). `backup.sh full --json`
#    prints exactly one JSON object on success — capture its backup_id.
#    On any failure `full` exits non-zero and prints no id, so BID stays
#    empty and we abort instead of verifying an older, pre-rotation
#    backup (verify requires --backup-id; there is no --label/--latest).
#    backup.sh reads MAID_DEPLOY_BACKUP_REMOTE (and the SSH key path)
#    from its OWN process env; a bare sudo does NOT load the instance
#    env, so source it with auto-export inside the maid-engine shell
#    (otherwise source_transport aborts: "MAID_DEPLOY_BACKUP_REMOTE is
#    not set").
BID="$(sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
        'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" full --instance <inst> --json' \
      | jq -r '.backup_id // empty')"
test -n "${BID}" || { echo "backup did not complete — aborting"; exit 1; }
export BID
sudo -u maid-engine --preserve-env=BID,MAID_SRC bash -c \
    'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance <inst> --backup-id "${BID}"' \
    || { echo "verify of ${BID} failed — aborting"; exit 1; }

# 5. Revoke the OLD public key on the destination, then delete the
#    .previous file:
sudo -u maid-engine rm "${KEY}.previous"

Section D — Admin API token¶

Important: the maid api generate-key/list-keys/revoke-key CLI operates on a cwd-relative JSON file (data/api_keys.json, DEFAULT_CLI_KEY_STORE_PATH). A running engine does not read that file live — its authoritative key store is DocumentStore-backed and is mutated only through the admin REST API. There is one narrow exception: at web-server startup the engine calls load_from_json_file(), which merges any CLI-generated keys from the cwd-relative data/api_keys.json into the in-memory store (best-effort; existing DocumentStore keys win on ID collision, and the imported keys are not written back to the DocumentStore). That import is (a) startup-only — a key the CLI writes while the server is running is not live until the next restart — and (b) cwd-dependent — it only fires if data/api_keys.json resolves relative to the server process's working directory. So the CLI is not a reliable live-rotation or live-inventory interface. To rotate a token on a running server use the REST API below, and treat GET /api/v1/admin/api-keys (not the CLI JSON file) as the authoritative live inventory.

BASE="http://127.0.0.1:8080/api/v1/admin"
CUR="<current admin token>"   # an X-API-Key that already has ADMIN

# 1. List current keys (find the key_id to revoke).
curl -fsS "${BASE}/api-keys" -H "X-API-Key: ${CUR}" \
  | jq '.keys[] | {key_id, name, is_active}'

# 2. Create the replacement key. `raw_key` is returned ONCE — capture
#    it now. Permissions are names (e.g. ADMIN, READ_PLAYERS, ...).
NEWKEY="$(curl -fsS -X POST "${BASE}/api-keys" -H "X-API-Key: ${CUR}" \
  -H 'Content-Type: application/json' \
  -d '{"name":"rotated-'"$(date -u +%Y%m%d)"'","permissions":["ADMIN"]}' \
  | jq -r '.raw_key')"

# 3. Smoke-test the new key against a real admin route, then deploy it
#    to every client (CI, monitoring, operator scripts).
curl -fsS "${BASE}/api-keys" -H "X-API-Key: ${NEWKEY}" >/dev/null && echo "new key OK"

# 4. Revoke the OLD key by key_id — effective immediately, no restart.
#    The API refuses to revoke the key you authenticate WITH, so make
#    this call using the NEW key.
curl -fsS -X DELETE "${BASE}/api-keys/<old-key-id>" -H "X-API-Key: ${NEWKEY}"

Section E — Admin/REST JWT signing key and `MAID_ADMIN_SECRET_KEY`¶

MAID signs two independent classes of JWT, and in the default configuration they do not share a key:

Admin-panel / REST JWTs are signed with the algorithm named by MAID_ADMIN_ALGORITHM (settings key admin.algorithm), which defaults to RS256. Under RS256 the signing material is an RSA key pair, not MAID_ADMIN_SECRET_KEY:
- private (signing) key: MAID_ADMIN_PRIVATE_KEY_PATH
- public (verify) key: MAID_ADMIN_PUBLIC_KEY_PATH If those paths are set, the PEM files at them are the keys. If they are unset, the engine auto-generates a fresh 2048-bit RSA pair on every start and never persists it — so under the default (paths unset) an admin JWT is already invalidated by any restart. To keep admin sessions valid across restarts you must pin the paths.
Web player-client JWTs (access + refresh) are always signed HS256 with MAID_ADMIN_SECRET_KEY (auth/core.py: get_player_auth_secret reuses admin.secret_key, and the algorithm is hard-coded HS256 regardless of admin.algorithm).

So MAID_ADMIN_SECRET_KEY: - always signs the web player-client tokens, and - signs the admin-panel/REST tokens only if you set MAID_ADMIN_ALGORITHM=HS256 (non-default). It is required at boot in production (the engine refuses to start without a secure 32+ char value) even under RS256, but in RS256 mode it is not the admin-JWT signing key.

Neither mechanism has a previous-key / grace slot (no MAID_ADMIN_SECRET_KEY_PREVIOUS, no second RSA key, and there is no MAID_SECURITY_SECRET_KEY* setting at all). Each rotation below is a flat swap + restart that invalidates the affected tokens at once.

E1 — rotate `MAID_ADMIN_SECRET_KEY`¶

Rotates the web player-client tokens (always), and the admin/REST tokens too only when MAID_ADMIN_ALGORITHM=HS256.

# 1. Generate a new key (min 32 chars; the engine refuses the insecure
#    default in production).
NEW_SECRET="$(python3 -c 'import secrets; print(secrets.token_urlsafe(64))')"

# 2. Install it in the instance env file (single key — no PREVIOUS slot).
sudo sed -i \
    -e "s|^MAID_ADMIN_SECRET_KEY=.*|MAID_ADMIN_SECRET_KEY=${NEW_SECRET}|" \
    /etc/maid/<inst>.env
# If the var wasn't present yet, append it:
grep -q '^MAID_ADMIN_SECRET_KEY=' /etc/maid/<inst>.env \
  || echo "MAID_ADMIN_SECRET_KEY=${NEW_SECRET}" | sudo tee -a /etc/maid/<inst>.env >/dev/null

# 3. Restart to load it (env is read only at process start; there is no
#    in-place/SIGHUP reload). At restart:
#      * web player-client sessions (access AND refresh) are invalidated
#        — those players must log in again;
#      * admin-panel / REST users must log in again ONLY if the instance
#        runs HS256 (MAID_ADMIN_ALGORITHM=HS256); under the default
#        RS256 their tokens are unaffected by this swap — rotate the RSA
#        pair in E2 for those;
#      * telnet sessions were dropped by the restart and re-authenticate
#        on reconnect regardless.
#    Because there is no grace window, schedule this during a
#    maintenance window (or accept the forced re-login for a compromise
#    rotation, where immediate invalidation is the goal).
sudo systemctl restart maid-engine@<inst>.service

E2 — rotate the admin/REST RSA signing key (default `RS256`)¶

Only needed when MAID_ADMIN_ALGORITHM is RS256 (the default) and you have pinned the key paths. If the paths are unset the engine regenerates the pair on every start, so a plain systemctl restart maid-engine@<inst>.service already rotates the admin-JWT key — nothing else to do.

# Generate a fresh 2048-bit RSA pair (PKCS#8 private, SPKI public — the
# formats the engine emits/consumes), keep .previous for rollback, and
# swap it into the pinned paths. Everything runs as the service account
# with the instance env sourced so the pinned paths resolve.
sudo -u maid-engine bash -c '
  set -a; . /etc/maid/<inst>.env; set +a
  test -n "${MAID_ADMIN_PRIVATE_KEY_PATH:-}" && test -n "${MAID_ADMIN_PUBLIC_KEY_PATH:-}" \
    || { echo "RSA key paths not pinned — a plain restart already rotates the ephemeral key"; exit 1; }
  umask 077
  openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \
      -out "${MAID_ADMIN_PRIVATE_KEY_PATH}.new"
  openssl pkey -pubout -in "${MAID_ADMIN_PRIVATE_KEY_PATH}.new" \
      -out "${MAID_ADMIN_PUBLIC_KEY_PATH}.new"
  cp -a "${MAID_ADMIN_PRIVATE_KEY_PATH}" "${MAID_ADMIN_PRIVATE_KEY_PATH}.previous"
  cp -a "${MAID_ADMIN_PUBLIC_KEY_PATH}"  "${MAID_ADMIN_PUBLIC_KEY_PATH}.previous"
  mv "${MAID_ADMIN_PRIVATE_KEY_PATH}.new" "${MAID_ADMIN_PRIVATE_KEY_PATH}"
  mv "${MAID_ADMIN_PUBLIC_KEY_PATH}.new"  "${MAID_ADMIN_PUBLIC_KEY_PATH}"
  chmod 0600 "${MAID_ADMIN_PRIVATE_KEY_PATH}"'

# Restart to load the new pair. There is no grace slot, so every
# admin/REST JWT signed by the old private key is rejected after the
# restart — admin users log in again. (Delete the .previous PEMs once
# the rollback window closes.)
sudo systemctl restart maid-engine@<inst>.service

Section F — Backup encryption key (NOT IMPLEMENTED)¶

There is no backup encryption key to rotate. backup.sh performs no encryption: MAID_BACKUP_ENCRYPTION_KEY / MAID_BACKUP_ENCRYPTION_KEY_PATH are read by nothing in the codebase (grep confirms the only occurrence was this doc). The backup pipeline provides:

Integrity, via per-file and per-manifest SHA-256 sidecars (MANIFEST.json.sha256), checked fail-closed by backup.sh verify.

Confidentiality in transit only when you use the ssh:// transport (MAID_DEPLOY_BACKUP_REMOTE=ssh://…, rsync-over-SSH). The local_path:// transport is a plain filesystem copy with no encryption. Backups are not encrypted at rest by MAID — any at-rest protection must come from the destination (LUKS, provider SSE, restricted filesystem permissions, etc.).

If you need encrypted-at-rest backups, apply it at the storage layer; do not add MAID_BACKUP_ENCRYPTION_KEY* to the env file expecting the script to honour it.

Recovery¶

For every rotation:

maid-admin doctor --phase runtime --instance <inst> returns all green.
/readyz returns 200.
maid-admin status --instance <inst> shows engine_state=running.
The next 30 minutes of the engine journal are CLEAN:

sudo journalctl -u maid-engine@<inst>.service \
    --since "30 minutes ago" --priority warning
# → no entries (or only known-benign warnings).

Section-specific verification:

A (DB): one full backup cycle completes (backup.sh full ... && backup.sh verify ...) AND no application_name LIKE 'maid-engine:%' row in pg_stat_activity still shows usename = maid_app_<inst>_v1 after Step 7.
B (AI): "${MAID_SRC}/.venv/bin/maid" dev test-ai 'hello, world' --provider <provider> returns a non-empty response, no 401/403 in the engine log.
C (Backup SSH): the post-rotation backup (captured id from full --json) passes verify --backup-id "${BID}".
D (Admin API): every client successfully calls a /api/v1/admin/* route (e.g. GET /api/v1/admin/api-keys) with the new token in the X-API-Key header; the old token returns 401.
E1 (MAID_ADMIN_SECRET_KEY): after the restart, pre-rotation web player-client JWTs are rejected and those users must log in again; admin-panel/REST users are forced to re-login too only if the instance runs HS256. A player token minted before the swap returns 401; new logins succeed against the new secret. (No grace window; invalidation is immediate at restart.)
E2 (admin/REST RSA pair): after the restart, admin-panel/REST JWTs signed by the old private key return 401 and those users log in again; new tokens verify against the new public key. (No grace window.)

Post-incident¶

Update ../deployment/security_checklist.md rotation-date table for the rotated secret.
File a ticket if any rotation step required a hand-edit not covered by this runbook (the runbook should be improved, not the operator's memory).
For a compromise rotation, file an incident ticket: what leaked, when, how it was detected, what the blast-radius window was, and whether downstream secrets need rotating too (e.g. a compromised admin token might have been used to read other env values from the admin API).
Confirm .previous env / key files are removed after their rollback windows close (mark a calendar reminder).

Limitations of current implementation¶

No maid-admin rotate-credentials priv-helper. The wrapper exposes the verb at packaging/admin/wrapper.py:446-452 but the helper-side handler is M1.3 work (see plan.md R10.A.10). Section A is hand-rolled until that ships.
No maid ops rotate-secret, and no grace/previous-key slot for either JWT key. MAID_ADMIN_SECRET_KEY is a single key with no MAID_ADMIN_SECRET_KEY_PREVIOUS, and the RS256 admin signing key (MAID_ADMIN_PRIVATE_KEY_PATH/MAID_ADMIN_PUBLIC_KEY_PATH) has no second/previous slot either; there is no dual-key grace window in the code (and no MAID_SECURITY_SECRET_KEY* setting exists). Plan §R3.3 (plan.md:1571-1574) proposes an OPS_SECRET_ROTATE verb and a grace pattern, but neither is wired up. Section E therefore does a flat swap + restart (immediate JWT invalidation) until that ships.
No maid-admin secrets list to query last-rotated timestamps. The 90-day cadence check in ../deployment/security_checklist.md currently relies on the operator's calendar, not an in-engine query.
No backup encryption / crypto-shred. backup.sh does not encrypt (see Section F); MAID_BACKUP_ENCRYPTION_KEY* is read by nothing. Crypto-shred for GDPR (./player_data_erasure.md) is unavailable for the same reason — erasure must delete records, not discard a key.
The maid api CLI is not a reliable live-rotation or live-inventory interface. It reads and writes a standalone JSON file (data/api_keys.json, cwd-relative). A running engine's authoritative store is DocumentStore-backed and is mutated only through the admin REST API (/api/v1/admin/api-keys). The engine imports the CLI JSON once, at startup (load_from_json_file(), best-effort, cwd-dependent, in-memory only), so CLI keys created while the server is up are not live until the next restart. Section D uses REST for this reason.

Escalation¶

Solo path: if any verification step fails, restore the env file from /etc/maid/<inst>.env.previous (Diagnostic Step 2) and sudo systemctl restart maid-engine@<inst>.service (there is no in-place reload). For Section A, also DROP ROLE maid_app_<inst>_v2 (if _v1 was already dropped in error, restore from your most recent backup per ./restore.md — the role definition is in the cluster, not the per-instance backup).
Hosting console URL: see ./escalation-contacts.md.template.
DNS registrar URL: see ./escalation-contacts.md.template.
Comms channel URL: see ./escalation-contacts.md.template.
Peer operator: see ./escalation-contacts.md.template.
If the engine cannot be brought back online within 30 minutes, notify players externally:

# Engine is down, so the in-game broadcast is unavailable and `maid ops
# maintenance` is an M9 stub. Post to the configured webhook(s). The
# webhook URL lives in the instance env file; `maid ops announce` reads
# MAID_BRIDGES_WEBHOOK_URLS from its OWN process environment, so source
# the env file with auto-export (`set -a`) — a plain `source` or a bare
# `--preserve-env` of a never-exported var forwards nothing:
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a;
   exec "${MAID_SRC}/.venv/bin/maid" ops announce --severity warn \
     --message "secret rotation rollback, ETA <ETA>"'

and post to the comms channel per ../deployment/player_comms.md.