Runbook: Two instances detected (leader lease contention)¶

Symptom¶

A second maid-engine@<inst>.service start is refused; engine journal:

LeaderLockHeld(holder_pid=<PID>, generation=<N>, lease_expires_at=<TS>)

systemctl status maid-engine@<inst>.service shows the second-comer in failed state with the above message in journal.
/readyz of the first instance is green and serving normally (lease holder is healthy).

The lease holder is doing the right thing. This runbook is for the operator who is trying to figure out why there is a second contender, and whether the second contender should win.

Detection¶

maid_leader_lease_contention_total increments.
LeaderLockHeld log line.
Doctor on the second host: maid-admin doctor --phase runtime --instance <inst> leader_lease_acquirable: fail.

Blast radius¶

Players affected: usually none — the first (legitimate) holder keeps serving.
Data at risk: only if a split-brain develops (two instances both writing). The leader-lease fencing in M1.2-A/B is designed to make this impossible, but cross-host lease contention is the trigger for validating that.
AI/external systems: irrelevant.

Prerequisites¶

Tools: maid-admin, ps, psql, systemctl, journalctl, optionally ssh (if the suspect holder is on another host).
Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units, which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) does not grant (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root). Plus maid_app_<inst> peer auth.
Env file: /etc/maid/<inst>.env.
Constants: LEASE_TTL_S = 30 (the engine renews the lease at half this interval; lease_expires_at must always be in the future for a live holder).
Audit access: tail -f /var/log/maid/ops-audit.jsonl.
Escalation: see ./escalation-contacts.md.template.

First 5 minutes (LITERAL commands)¶

# 1. Read the lease state directly from the source of truth (PG).
#    Schema reference: packaging/postgres/migrations/0001_leader_lock.sql.
#    Columns: singleton (bool, always true), holder_pid, generation,
#    leased_at, lease_expires_at. There is intentionally NO holder_host
#    column — the engine identifies the holding host via pg_stat_activity
#    join below (R10.B.6 / R11.B.2).
sudo -u maid_app_<inst> psql -d maid_<inst> -c "
  select holder_pid, generation,
         leased_at, lease_expires_at,
         lease_expires_at - now() as ttl_remaining
  from leader_lock;"

# 2. Identify the holder host/process by joining pg_stat_activity.
#    The engine's asyncpg pool sets application_name='maid-engine:<instance>'
#    (packages/maid-engine/src/maid_engine/persistence/lease.py:23-24,132;
#    stale_writer.py:79-95 — bare "maid-engine" is the legacy fallback only).
sudo -u maid_app_<inst> psql -d maid_<inst> -c "
  select sa.pid, sa.application_name, sa.client_addr, sa.backend_start,
         sa.state
  from pg_stat_activity sa
  join leader_lock ll on ll.holder_pid = sa.pid
  where sa.application_name LIKE 'maid-engine:%';"
HOLDER_PID=$(sudo -u maid_app_<inst> psql -d maid_<inst> -tAc "select holder_pid from leader_lock")
HOLDER_ADDR=$(sudo -u maid_app_<inst> psql -d maid_<inst> -tAc "
  select coalesce(host(client_addr), 'local') from pg_stat_activity
  where pid = $HOLDER_PID;")
echo "holder pid=$HOLDER_PID client_addr=$HOLDER_ADDR"
echo "this host=$(hostname --fqdn)"

# 3. If the holder is on this host (client_addr empty/local AND the PID
#    is a local process), is the process actually alive?
if [ -z "$HOLDER_ADDR" ] || [ "$HOLDER_ADDR" = "local" ]; then
  ps -o pid,user,etime,cmd -p "$HOLDER_PID" || echo "PID $HOLDER_PID is GONE"
fi

# 4. Capture state of the rejected second-comer
sudo journalctl -u maid-engine@<inst>.service -n 200 \
  > ./incident-rejected-$(date -u +%Y%m%dT%H%M%SZ).log

Investigation — decision tree¶

lease_expires_at > now()? (lease still valid)
└── yes (lease is fresh, holder is alive and healthy)
    │
    ├── pg_stat_activity join shows client_addr == local AND PID alive?
    │     => Legit holder on this box. The second-comer should NOT exist.
    │        Stop the second; see "Stop the second" below.
    │
    ├── pg_stat_activity join shows client_addr == local AND PID GONE?
    │     => Zombie lease (rare; should self-heal in <LEASE_TTL_S).
    │        Wait 30s, retry. See "Zombie lease" below.
    │
    └── client_addr is a different host (or pg_stat_activity has no row
        for holder_pid at all)
          => Holder is on ANOTHER box (or PG can no longer see it). This
             is the DISASTER scenario. See "Cross-host holder" below.

lease_expires_at <= now() (lease is stale / expired)
└── Either the holder just died (takeover predicate will fire on the
    next start attempt), or NTP skew. Wait 5-10s and retry. If still
    stale, the lease is abandoned and the next start should win the
    takeover atomically.

Mitigation¶

"Stop the second" — legit holder, accidental second start¶

The first instance is fine. Just stop the one you started by mistake:

sudo systemctl stop maid-engine@<inst>.service  # the one you just started
# (it already failed; this just cleans the unit state)
sudo systemctl reset-failed maid-engine@<inst>.service

Confirm only one is running across the cluster:

sudo -u maid_app_<inst> psql -d maid_<inst> -c "
  select holder_pid, generation, leased_at, lease_expires_at from leader_lock;"

"Zombie lease" — holder PID is gone but lease not expired¶

This is a process-death race. The lease will expire on its own within LEASE_TTL_S (30 s). Just wait and retry:

sleep 35
sudo systemctl restart maid-engine@<inst>.service
sudo journalctl -u maid-engine@<inst>.service -n 50

If repeatable, something is preventing the watchdog from releasing the lease on shutdown — file a ticket and investigate.

"Cross-host holder" — split-brain risk¶

The lease says it's held by a connection whose client_addr is a different host (or pg_stat_activity has no row for holder_pid at all, meaning the holder is not currently connected to this PG). Possibilities: - That other box is the real production instance and you are trying to start a duplicate on the wrong host. STOP. Do not start the engine here. - DNS/hostname misconfiguration: this is actually the same box, but network routing makes the holder appear remote. - A real disaster: the original box went down, the lease did NOT expire (clock skew, partial network), and you are bringing up a recovery host.

Procedure:

# 1. Identify what host the lease's holder PID is connected from
sudo -u maid_app_<inst> psql -d maid_<inst> -c "
  select sa.pid, host(sa.client_addr) as client_host, sa.backend_start
  from pg_stat_activity sa
  join leader_lock ll on ll.holder_pid = sa.pid
  where sa.application_name LIKE 'maid-engine:%';"
echo "this host : $(hostname --fqdn)"

# 2. Try to reach the apparent holder host (substitute the client_host
#    printed by the query above; left here as <HOLDER_HOST>)
ping -c 2 <HOLDER_HOST> || echo "unreachable"
ssh <HOLDER_HOST> "systemctl status maid-engine@<inst>.service --no-pager" \
  2>/dev/null || echo "ssh failed"

Holder host reachable AND has an active engine: that is the legitimate production instance. You are on the wrong box. Do NOT start the engine here. End of incident.
Holder host is unreachable: go to ./RB18_host_loss_dr.md. The lease will expire once lease_expires_at <= now() (within LEASE_TTL_S of the last successful heartbeat). Do NOT attempt manual lease tampering unless RB18 explicitly instructs you to — fencing-by-generation in M1.2-A guarantees safety only if you let the takeover predicate run.

Never manually DELETE FROM leader_lock or UPDATE leader_lock from a SQL client. The takeover predicate (lease_expires_at < now() on the canonical schema in packaging/postgres/migrations/0001_leader_lock.sql) plus generation bump is the only safe path to a new holder.

Recovery¶

sudo -u maid_app_<inst> psql -d maid_<inst> -c "select * from leader_lock" shows exactly one row with lease_expires_at in the future and the expected holder_pid (matched to a local engine via pg_stat_activity).
maid-admin doctor --phase runtime --instance <inst> all green.
/readyz returns 200 on the legitimate holder.

Post-incident¶

File ticket with: which scenario fired (accidental second / zombie / cross-host), root cause, time to recover.
If cross-host: review hostname configuration, deploy automation, and HA assumptions. Most cross-host contention in single-host MAID is an operator mistake; if you intentionally run multi-host, document the active host explicitly.
Update this runbook if any step did not work as written.

Escalation¶

Solo path for accidental-second: stop the new one, reset-failed.
Solo path for zombie: wait LEASE_TTL_S, retry.
Solo path for cross-host: do NOT start the engine here unless RB18 says so.
Hosting console URL: see ./escalation-contacts.md.template.
DNS registrar URL: see ./escalation-contacts.md.template.
Comms channel URL: see ./escalation-contacts.md.template.
Peer operator: see ./escalation-contacts.md.template.
If you cannot determine which holder is authoritative within 30 min, go to ./RB18_host_loss_dr.md and rebuild on a known-clean host from a verified backup. Losing the lease race on the wrong host is recoverable; double-writing from two engines may not be.

Alternate alerting path (no Prometheus)¶

Operators without a Prometheus / Alertmanager stack get the same coverage via the maid-doctor timer + alert-script pair shipped in PR-C. See deploy/monitoring/doctor-alert.sh and packaging/systemd/maid-doctor@.timer.

Enable per instance:

sudo systemctl enable --now maid-doctor@<inst>.timer

On every 5-minute tick the timer runs scripts/maid-doctor.sh --phase runtime --json, pipes the result into doctor-alert.sh, which then:

Writes /var/lib/node_exporter/textfile_collector/maid_doctor.prom (metrics maid_doctor_last_run, maid_doctor_last_status, maid_doctor_failed_checks_total) so any future scraper picks up the most recent doctor verdict without re-running it.
On warn or fail, emails the on-call address read from /etc/maid-engine/<inst>/oncall.env (ONCALL_EMAIL).

This is the default monitoring path for solo home-lab installs; Prometheus is the optional add-on for shops that want graphs/history.