MAID Metrics Catalog¶

Single source of truth for every Prometheus metric the deployment stack expects. Maintained alongside deploy/monitoring/alerts.yml and deploy/monitoring/grafana/maid-dashboard.json. If you add or rename a metric in the engine, update this file in the same commit.

Plan citations: §R3.9 (monitoring quickstart), §R3.14 (restore-drill metrics), §R10.A (alerting baseline), PR-C (metrics critic).

Conventions¶

All engine-side metrics are auto-prefixed with maid_ by observability/metrics.py:_metric_name. Pack-scoped metrics additionally carry the pack name, e.g. maid_classic_rpg_*.
*_total is the Prometheus counter convention; *_seconds / *_bytes are the unit suffixes per OpenMetrics; rolling gauges use no suffix or a descriptive one (_state, _depth).
Shell helpers (deploy/scripts/backup.sh, deploy/scripts/restore.sh) write metrics via the node_exporter textfile collector in atomic temp+rename style.
kind label distinguishes manual vs automated invocations of restore-drill (R3.14): alerts and dashboard panels MUST select {kind="automated"} so a manual drill does not reset the staleness clock without operator acknowledgement.

Status legend¶

live — emitted today by the cited source.
wave-2 — referenced by parked alert / dashboard panel; owning milestone identified. Replace vector(0) placeholders with the real expr once the metric ships.
exporter — supplied by a third-party Prometheus exporter (node_exporter / postgres_exporter) and not produced by the engine itself.

Live engine metrics¶

Metric	Type	Source	Used by alerts	Used by dashboard panels
`maid_tick_duration_seconds` (histogram)	histogram	`core/engine.py`	—	—
`maid_system_tick_duration_seconds`	histogram (labels `pack_name`, `system_name`)	`observability/metrics.py`	—	—
`maid_tick_loop_healthy`	gauge	`core/engine.py`	—	—
`maid_entities_total`	gauge	`observability/hooks.py`	—	—
`maid_connections_total`	gauge	`observability/hooks.py` (engine-level rollup)	—	504 Open connections
`maid_ticks_total`	counter	`core/engine.py`	—	—
`maid_events_total`	counter (label `event_domain`)	`core/events.py`	—	—
`maid_net_connections_active`	gauge (label `protocol`)	`net/server.py`, `net/web/server.py`	—	—
`maid_net_bytes_sent_total`	counter (label `protocol`)	`net/server.py`, `net/web/server.py`	—	—
`maid_net_bytes_received_total`	counter (label `protocol`)	`net/server.py`, `net/web/server.py`	—	—
`maid_net_messages_sent_total`	counter (label `protocol`)	`net/server.py`, `net/web/server.py`	—	—
`maid_net_messages_received_total`	counter (label `protocol`)	`net/server.py`, `net/web/server.py`	—	—
`maid_db_query_duration_seconds`	histogram (labels `operation`, `collection`)	`storage/document_store.py`	—	—
`maid_db_operations_total`	counter (labels `operation`, `collection`, `status`)	`storage/document_store.py`	—	—
`maid_db_connections_active`	gauge	`storage/document_store.py`	—	—
`maid_memory_memories_decayed_total`	counter	`ai/metrics.py`	—	—
`maid_ai_rate_limit_allowed_total`	counter	`ai/rate_limiter.py`	—	—
`maid_ai_rate_limit_utilization`	gauge	`ai/rate_limiter.py`	—	—
`maid_ai_tokens_total`	counter (labels `provider`, `model`, `token_type`)	`ai/providers/base.py`	—	—
`maid_ai_streaming_requests_total`	counter (labels `provider`, `model`, `status`)	`ai/providers/base.py`	—	—
`maid_ai_streaming_duration_seconds`	histogram (labels `provider`, `model`)	`ai/providers/base.py`	—	—
`maid_ai_cache_tokens_total`	counter (labels `provider`, `model`, `cache_type`)	`ai/providers/base.py`	—	—
`maid_memory_usage_bytes`	gauge	`observability/hooks.py`	—	—
`maid_observability_errors_total`	counter	`observability/safe_observe.py`	—	—
`maid_http_requests_total`	counter	`observability/middleware.py`	—	—
`maid_http_request_duration_seconds`	histogram	`observability/middleware.py`	—	—
`maid_commands_total`	counter	`commands/registry.py`	—	602 Command rate
`maid_command_duration_seconds` (histogram)	histogram	`commands/registry.py`	—	—
`maid_persistence_dirty_entities`	gauge	`persistence/scheduler.py`	MaidSaveQueueGrowing	201 Save queue depth
`maid_persistence_save_duration_seconds`	histogram	`persistence/scheduler.py`	—	—
`maid_persistence_save_errors_total`	counter	`persistence/scheduler.py`	MaidPersistenceSaveErrors	203 Save errors (rate)
`maid_ai_cost_total_usd`	gauge	`observability/ai_metrics.py`	—	—
`maid_ai_cost_by_model_usd`	gauge	`observability/ai_metrics.py`	—	—
`maid_ai_cost_rolling_usd`	gauge	`observability/ai_metrics.py`	—	—
`maid_ai_tracked_tokens_total`	gauge	`observability/ai_metrics.py`	—	—
`maid_ai_player_actions_total`	gauge	`observability/ai_metrics.py`	—	—
`maid_ai_player_violations_total`	gauge	`observability/ai_metrics.py`	—	—
`maid_ai_rate_limit_denials_total`	counter	`ai/rate_limiter.py`	MaidAIRateLimiterDenying	—
`maid_ai_requests_total`	counter	`ai/providers/base.py`	—	402 AI request rate
`maid_ai_request_duration_seconds` (histogram)	histogram	`ai/providers/base.py`	—	403 AI latency p99
`maid_ai_request_cost_usd_total`	counter	`ai/providers/base.py`	—	404 AI cost (USD/hour)
`maid_ai_circuit_breaker_state`	gauge (`provider` label; 0=closed/1=open/2=half)	`ai/circuit_breaker.py`	MaidAICircuitBreakerOpen	401 AI breaker state
`maid_ai_circuit_breaker_state_indicator`	gauge (`provider`, `state` labels; 1 for active state, 0 otherwise)	`ai/circuit_breaker.py`	MaidAICircuitBreakerOpen	401 AI breaker state
`maid_ai_circuit_breaker_trips_total`	counter	`ai/circuit_breaker.py`	—	—
`maid_ai_circuit_breaker_failures_total`	counter	`ai/circuit_breaker.py`	—	—
`maid_ai_circuit_breaker_recovery_attempts_total`	counter	`ai/circuit_breaker.py`	—	—
`maid_ai_kill_switch_tripped`	gauge (0/1)	`ai/kill_switch.py`	—	—
`maid_ai_kill_switch_reasons_count`	gauge	`ai/kill_switch.py`	—	—
`maid_ai_budget_exhausted_total`	counter (`scope` label: global, player)	`ai/budget.py` via `ai/rate_limiter.py`	—	—
`maid_ai_queue_depth`	gauge (`queue` label)	`ai/queue_metrics.py` (content packs call)	—	—
`maid_ai_tick_isolation_violations_total`	counter (`mode` label: warn, reject)	`ai/tick_isolation.py`	—	—
`maid_memory_extractions_total`	counter	`ai/metrics.py`	—	—
`maid_memory_memories_stored_total`	counter	`ai/metrics.py`	—	—
`maid_memory_retrievals_total`	counter	`ai/metrics.py`	—	—
`maid_memory_extraction_duration_ms`	histogram	`ai/metrics.py`	—	—
`maid_memory_extraction_queue_depth`	gauge	`ai/metrics.py`	—	—
`maid_memory_cache_hit_rate`	gauge	`ai/metrics.py`	—	—
`maid_memory_gossip_exchanges_total`	counter	`ai/metrics.py`	—	—
`maid_memory_consolidations_total`	counter	`ai/metrics.py`	—	—

Live shell-helper metrics (textfile collector)¶

⚠ M9-pending packaging limitation. The _success / _status(=1) / restore-drill success gauges below are written by backup.sh / restore.sh, but the hardened maid-backup@.service and maid-restore-drill@.service run with ProtectSystem=strict and have no write path to /var/lib/node_exporter/textfile_collector, so those success writes silently no-op today. Only the root-owned maid-backup-failure@ / maid-backup-prune-failed@ hooks currently emit a metric (the 0/failure series). Until the M9 packaging fix adds the textfile dir to ReadWritePaths=, treat the success series as absent and confirm backups via the OPS_BACKUP_COMPLETE audit rows and the failure alerts.

Metric	Type	Source	Used by alerts	Used by dashboard panels
`maid_backup_last_success_timestamp_seconds`	gauge	`deploy/scripts/backup.sh`	MaidBackupFailed	301 Backup: last success
`maid_backup_last_status`	gauge	`deploy/scripts/backup.sh`	—	—
`maid_backup_last_restore_drill_timestamp` (label `kind`)	gauge	`deploy/scripts/restore.sh` `--report-metric`	MaidRestoreDrillStale, MaidRestoreDrillCritical	302 Restore drill: last run
`maid_backup_last_restore_drill_success` (label `kind`)	gauge	`deploy/scripts/restore.sh` `--report-metric`	MaidRestoreDrillFailed	—

Live exporter metrics (third-party)¶

These are not engine-emitted; they come from node_exporter / postgres_exporter and so always carry the exporter's own label set.

Metric	Source	Used by alerts	Used by dashboard panels
`up{job="maid-engine"}`	Prometheus self-scrape	MaidEngineDown	—
`up{job="redis"}`	redis_exporter	MaidRedisDown	—
`up{job="postgres"}`	postgres_exporter	MaidPostgresDown	—
`node_systemd_unit_start_time_seconds{name=~"maid-engine@.*\.service"}`	node_exporter systemd collector	MaidOOMLoop	503 Process starts

Wave-2-pending metrics¶

Each row below corresponds to a parked alert (commented out with # WAVE-2-PENDING in alerts.yml) and/or a parked dashboard panel (rendered as a yellow "PENDING — Wave 2" sentinel). When the owning milestone lands the metric, restore the alert and replace the sentinel panel's vector(0) target with the real expr.

Metric	Type	Owning milestone	Parked alert(s)	Parked dashboard panel(s)
`maid_engine_ready`	gauge	m12b-engine-wiring	MaidEngineNotReady	101
`maid_leader_lock_generation`	counter	m12b-engine-wiring	MaidLeaderLockStolen	102
`maid_heartbeat_last_renewed_timestamp`	gauge	m12b-engine-wiring	MaidHeartbeatStale	103
`maid_tripswitch_tripped`	gauge	m12b-engine-wiring	MaidTripswitchTripped	104
`maid_persistence_save_queue_oldest_age_seconds`	gauge	m12b-engine-wiring	—	202
`maid_lease_ttl_remaining_seconds`	gauge	m12b-engine-wiring	—	204
`maid_backup_bytes_transferred_total`	counter	backup.sh extension	—	303
`maid_backup_duration_seconds`	gauge	backup.sh extension	—	304
`maid_disk_free_bytes` / `maid_disk_total_bytes` (label `mount`)	gauge	m9-disk-monitor	MaidDiskFullData, MaidDiskFullWAL	501, 502
`maid_sessions_connected`	gauge	m12b-engine-wiring	—	601
`maid_command_errors_total`	counter	m12b-engine-wiring	—	603
`maid_migrations_failed_total`	counter	m12b-engine-wiring	MaidMigrationFailed (parked group)	—
`maid_preflight_failed_total`	counter	m12b-engine-wiring	MaidPreflightFailed (parked group)	—
`maid_login_rate_limited_total`	counter	m8-security	MaidLoginRateLimitTriggered	—

Interim coverage notes¶

Where a wave-2 metric blocks an operationally important alert today, operators can deploy a node_exporter-based rule until the engine ships its native metric:

Disk fullness — until maid_disk_free_bytes ships, use:

- alert: MaidDiskFullDataInterim
  expr: |
    node_filesystem_avail_bytes{mountpoint="/var/lib/maid/data"}
      / node_filesystem_size_bytes{mountpoint="/var/lib/maid/data"}
      < 0.10
  for: 5m
  labels: { severity: critical, component: storage }
  annotations:
    summary: "MAID data mount on {{ $labels.instance }} has <10% free"
    runbook_url: "https://docs.maid-engine.example/runbooks/disk_full.md"

Replace the mountpoint selector with the actual data / WAL paths for your deployment.

Process readiness — until maid_engine_ready ships, the combination of up{job="maid-engine"} == 1 and a non-empty recent maid_tick_loop_healthy == 1 provides a usable approximation; node_systemd_unit_state{name=~"maid-engine@.*", state="active"} is the most authoritative proxy on a single node.

Adding a new metric¶

Register the metric in the relevant packages/maid-engine/src/maid_engine/observability/ module (or in the owning subsystem) using meter.create_counter / create_gauge / create_histogram. The maid_ prefix is added automatically — pass the un-prefixed name (e.g. persistence_dirty_entities).
If the metric is consumed by an alert or panel, add a row to the table above in the same commit.
If the metric supersedes a wave-2-pending entry, remove the pending row and un-park the corresponding alert / panel.
Update deploy/monitoring/alerts.yml and deploy/monitoring/grafana/maid-dashboard.json together so the catalog never drifts from the live config.