Skip to content

MAID Service Level Objectives (Single-Box, Honest)

This is single-box. HA is out of scope for M1. All RPO/RTO numbers below assume one production host, WAL archiving to a second location, and restore-from-backup as the failure mode. The RTO numbers assume the operator is reachable within the page response window. If your on-call response time is N minutes, add N to every RTO.

Targets

Metric Target Notes
Availability SLO 99.0% monthly (≈7h 18m/mo allowance) Single-box; planned maintenance counted. Excludes declared maintenance windows announced via maid ops announce.
Tick lag p99 < 2× configured tick interval (e.g. < 500ms at tick_rate=4/s) Measured over 5-minute windows.
Command response p99 (non-AI) < 250ms Excludes commands that invoke an external LLM.
Command response p99 (AI) < 5s Tracked separately; AI provider outage does not break the non-AI SLO.
RPO — durable game state, critical class ≤ 5s Currency, inventory transfer, quest completion, builder commit. See durability matrix.
RPO — durable game state, normal class ≤ save_interval (default 300s / 5 min) HP/MP, combat state, memory, relationships, NPC autonomy. Set MAID_PERSISTENCE_SAVE_INTERVAL lower than the 300s default if a tighter normal-class RPO is required.
RPO — DB on disk failure ≤ 5 min only if continuous WAL archiving is configured (see note) With the shipped defaults this RPO is not met: archive_mode = off in postgresql.tuning.conf.template (archive setup deferred to M1.4), so recovery is limited to the last pg_basebackup full. Achieving ≤ 5 min is a conditional operator responsibility — enable archive_mode = on + a working archive_command + off-host WAL shipping.
RTO — process crash ≤ 60s systemd restart + integrity scan + world reload.
RTO — disk failure ≤ 30 min Provision box + restore from off-host backup + WAL replay. Documented procedure with quarterly drill.
RTO — failed migration ≤ 15 min Rollback to prior version + replay backward-compat migrations down. See migration policy and docs/runbooks/failed_migration.md.
Backup success rate SLO 99.5% (monthly) Alert on 2 consecutive failures.
Restore test cadence Monthly automated drill (≤ 35 days) + 1× per quarter manual promote-to-staging (≤ 92 days) The automated drill restores the latest backup to a scratch host and runs an integrity scan. The quarterly drill additionally boots a full staging engine against the restored data. A miss on either invalidates the disk-failure RTO/RPO claim.

Monitoring queries

Each row above maps to at least one of:

  • a PromQL query against the metrics scraped from MAID_OBSERVABILITY_INTERNAL_PORT (default 9090, loopback-bound), and
  • an ops_audit.jsonl SQL query for events not surfaced as metrics (typically using jq -s + sqlite-utils memory or DuckDB on the file).

The audit ledger lives at /var/log/maid/ops-audit.jsonl (per plan.md R10.B.11 — ledger is observational/audit only).

Availability (target: ≥ 99.0% monthly)

# Fraction of scrape windows where the engine was up over 30 days.
avg_over_time(up{job="maid"}[30d])

Burn-rate alert:

(1 - avg_over_time(up{job="maid"}[1h])) > (1 - 0.99) * 14.4

Tick lag p99 (target: < 2 / tick_rate seconds)

⚠️ M9-PENDING. maid_tick_duration_seconds_bucket and maid_tick_rate are not yet registered metrics. The current observability registry (packages/maid-engine/src/maid_engine/observability/metrics.py) auto-prefixes maid_ to registered names; the tick histogram lands as part of M9. Until M9 lands the queries below will silently evaluate to nothing — any alert built on them WILL NEVER FIRE. Until then, derive tick lag from the tick.completed event class in /var/log/maid/ops-audit.jsonl.

# Tick duration p99 expressed as multiples of the configured tick interval.
histogram_quantile(0.99, rate(maid_tick_duration_seconds_bucket[5m]))
  * maid_tick_rate

Alert when this value exceeds 2.0 for 10 minutes.

Command response p99 (target: non-AI < 250ms; AI < 5s)

⚠️ M9-PENDING. maid_command_duration_seconds_bucket (with the ai="true|false" label) is not yet a registered metric; lands with M9. The queries below will silently evaluate to nothing until then. Until M9 lands, query the command.completed event class in the audit ledger.

# Non-AI commands.
histogram_quantile(0.99,
  rate(maid_command_duration_seconds_bucket{ai="false"}[5m]))

# AI commands.
histogram_quantile(0.99,
  rate(maid_command_duration_seconds_bucket{ai="true"}[5m]))

RPO — critical-class writes (target: ≤ 5s)

⚠️ M9-PENDING. maid_persistence_save_queue_oldest_age_seconds is not yet a registered gauge; lands as part of the persistence instrumentation in M9. The PromQL below will silently evaluate to nothing until then. Until M9, derive from the critical_write.flushed audit-ledger event below.

# Oldest unflushed entity in the persistence queue, in seconds.
max(maid_persistence_save_queue_oldest_age_seconds)

-- Critical-class flush latency from the audit ledger.
SELECT
  json_extract(payload, '$.class')          AS write_class,
  max(json_extract(payload, '$.elapsed_ms')) AS p100_elapsed_ms
FROM read_json_auto('/var/log/maid/ops-audit.jsonl')
WHERE event = 'critical_write.flushed'
  AND timestamp > now() - INTERVAL 1 HOUR
GROUP BY 1;

Alert when maid_persistence_save_queue_oldest_age_seconds > 5 for the critical class, or > save_interval for the normal class.

RPO — DB on disk failure (target: ≤ 5 min, conditional)

⚠️ NOT ACHIEVED BY DEFAULT. The shipped postgresql.tuning.conf.template sets archive_mode = off (continuous WAL archiving is deferred to M1.4), so with defaults the recovery point is the last successful pg_basebackup full — not 5 minutes. The ≤ 5 min target requires the operator to (1) set archive_mode = on + a working archive_command (e.g. deploy/scripts/backup.sh wal-archive), (2) restart the cluster (archive_mode is restart-only), and (3) ship WAL off-host. See packaging/postgres/README.md ("Enabling WAL archiving").

⚠️ M9-PENDING (metrics/probe). Even once WAL archiving is enabled, maid_pg_wal_last_archived_timestamp_seconds is not yet a registered gauge and the wal_archive_check doctor probe is an M9 stub; the Postgres exporter integration lands in M9. The PromQL below will silently evaluate to nothing until then. Until M9, parse pg_stat_archiver.last_archived_time directly via the systemd backup job's log.

# WAL archive age. Should not exceed 5 minutes.
time() - max(maid_pg_wal_last_archived_timestamp_seconds)

Cross-check the readiness gate: /readyz (observability port 9090) is specified to flip red when WAL archive lag exceeds 2× the disk-failure RPO (plan.md R4.10). The engine-side maid ops doctor --json runtime check that would surface wal_archive_check is an M9 stub today (prints NOT YET IMPLEMENTED); until it lands, rely on /readyz and the PromQL query above.

RTO — process crash (target: ≤ 60s)

-- Median crash-to-ready interval for the last 30 days.
SELECT median(
  json_extract(payload, '$.ready_at_epoch')
  - json_extract(payload, '$.crash_at_epoch')
) AS median_recovery_seconds
FROM read_json_auto('/var/log/maid/ops-audit.jsonl')
WHERE event = 'engine.recovered_from_crash'
  AND timestamp > now() - INTERVAL 30 DAY;

Backup success rate (target: ≥ 99.5% monthly)

⚠️ M9-PENDING. maid_backup_last_success_timestamp_seconds and maid_backup_consecutive_failures are not yet registered metrics; they land with the backup-job instrumentation in M9. The PromQL below will silently evaluate to nothing until then. Until M9, parse /var/log/maid/backups.jsonl (written by deploy/scripts/backup.sh) and alert via the backup.completed event class.

# Seconds since the last successful backup. Alert > 25h.
time() - max(maid_backup_last_success_timestamp_seconds)

# Consecutive failure count. Alert at 2.
max(maid_backup_consecutive_failures)

Restore drill freshness (target: ≤ 45 days automated + ≤ 100 days alert)

⚠️ M9-PENDING. maid_backup_last_restore_drill_timestamp (with the kind="automated|manual" label) is not yet a registered gauge; it lands with the restore-drill instrumentation in M9. Note the metric name has no _seconds suffix — it is a Unix epoch timestamp written by deploy/systemd/maid-restore-drill.service to /var/lib/maid/metrics/restore-drill.prom via the Prometheus textfile collector (see plan.md R3.14). The PromQL below will silently evaluate to nothing until M9 lands. Until then, emit the timestamps from the drill scripts themselves and parse them out of the audit ledger via the restore_drill.completed event class.

# Seconds since the last automated monthly restore drill.
# WARN at 45 days (45 * 86400 = 3888000).
# ALERT at 100 days (100 * 86400 = 8640000) — see plan.md R3.14, alert
# rule MaidRestoreDrillStale at plan.md line ~1149.
time() - max(maid_backup_last_restore_drill_timestamp{kind="automated"})

# Seconds since the last manual quarterly promote-to-staging drill.
# Alert when this exceeds 100 days (100 * 86400 = 8640000).
time() - max(maid_backup_last_restore_drill_timestamp{kind="manual"})

Both gauges feed the disk-failure RPO/RTO claim. If either is stale or has no sample, the claim is invalid. The runbook for this state is docs/runbooks/restore.md. The monthly drill is a no-touch cron that runs restore.sh verify-latest --instance <inst> --staging --no-promote; the quarterly drill is a human-led full promote-to-staging exercise.

What "honest" means here

  • No HA, no failover. A second box exists only as a WAL/backup target. Disk-failure recovery requires provisioning a fresh host or repurposing the backup host.
  • The RTO numbers exclude operator response time. A page sent to a sleeping operator at 03:00 does not satisfy a 60s RTO. Page-to-ack time belongs to your incident-response SLA, not to MAID.
  • A failed restore drill invalidates the disk-failure RPO/RTO. Treat a miss on either the monthly automated drill or the quarterly manual drill as an SLO breach until the next drill passes.
  • The "save queue oldest age" metric is the single best leading indicator for an RPO violation. Wire its alert before any cosmetic dashboard.