Skip to content

Operational Observability — Design Document

Version: 3.3 Status: Final Author: Platform/SRE Architecture Last Updated: 2025-07-17 Package: maid-engine (core), all content packs (instrumentation) Inputs: Original draft, co-writer expansion, two Devil's Advocate critiques, six expert reviews, four advisor reviews

Table of Contents

  1. Executive Summary
  2. Problem Statement & Current State
  3. Architecture Overview
  4. Metrics Design
  5. AI Cost Monitoring
  6. Structured Logging
  7. Distributed Tracing
  8. SLO/SLI Definitions
  9. Dashboard & Alerting
  10. Integration with Existing Profiling
  11. Development Mode
  12. Cardinality Management
  13. Health Checks
  14. Security & Privacy
  15. Runbooks & Operational Readiness
  16. Testing Strategy
  17. Performance Impact
  18. Implementation Plan
  19. Open Questions & Future Work
  20. Design Decisions Log

1. Executive Summary

MAID has zero external observability. There are no Prometheus metrics, no structured logs, no distributed traces. The internal profiling system (@profile, @memory, @timing) collects rich data — tick durations, query latencies, network I/O, memory allocations — but this data is only accessible through in-game admin commands and never leaves the process. The admin dashboard streams metrics over WebSocket, but only to the React admin UI.

For production deployment this is a critical gap. Operators cannot monitor health via orchestrator probes, track AI spending (a runaway NPC conversation can silently exhaust budgets), set up alerts for tick overruns or memory leaks, or ship logs to aggregation systems without fragile regex parsing.

This document designs a comprehensive observability layer:

  1. Structured logging via structlog — JSON output, correlation IDs, context propagation, adaptive sampling (with error/slow-path bypass), privacy-aware (HMAC anonymization, whitelist-based command redaction), separate audit/operational sinks
  2. Prometheus metrics via /metrics on a dedicated internal port (default 9090) — game metrics, AI cost metrics with mandatory intent label for feature attribution, quest generation pipeline metrics, NPC autonomy metrics, infrastructure metrics with bounded cardinality, background-rendered scrape cache, and operator-tunable cardinality guardrails
  3. AI cost monitoring (the #1 priority) — per-provider/per-model cost in USD with per-feature intent attribution, budget alerting with per-player spend monitoring, per-player attribution via event-driven AICostTracker, cancellation-safe streaming cost tracking via CompletionChunk
  4. Distributed tracing via OpenTelemetry (optional dependency) with configurable tracing modes (minimal/default/verbose) and head-based sampling
  5. Health check endpoints (/healthz, /readyz, /livez) on internal port with cached DB checks
  6. SLO/SLI definitions with multi-window burn-rate alerting and error budgets
  7. Grafana dashboards and Prometheus alerting rules (operator-configurable via templates) for day-one operational readiness
  8. Runbooks shipped for critical alerts, with on-call workflow and single-operator guidance
  9. Error tracking via optional Sentry SDK with game-specific context

The design instruments existing code paths with minimal changes — primarily by bridging the data already collected by ProfileManager, MetricsCollector, AuditLogStore, and RateLimiter into standard observability formats.

Required dependencies: structlog, prometheus-client (both lightweight, zero transitive deps). Optional dependencies: opentelemetry-sdk (tracing), sentry-sdk (error tracking), psutil (CPU/VMS metrics). Estimated effort: 10–14 weeks across 3 phases.

2. Problem Statement & Current State

2.1 Internal Profiling System

The maid_engine.profiling package provides session-based profiling via admin commands. It is not an observability system — it requires manual activation (@profile start), stores data in-process, and has no external export.

Collector Data Collected Access
TickCollector Per-tick duration, per-system timing, slow tick detection (>250ms) @timing
QueryCollector Per-query duration, operation type, collection, slow queries (>100ms) @profile
NetworkCollector Per-connection bytes, message counts by type, protocol breakdown @profile
MemoryCollector tracemalloc snapshots, allocation by type/module, leak detection @memory
ProfileManager Orchestrates sessions with auto-stop. Disabled by default. @profile start/stop

The profiling system computes percentiles, per-system breakdowns, and slow-operation detection — exactly the aggregations needed for Prometheus. However, because profiling is session-based and disabled by default, direct Prometheus instrumentation in the tick loop is the correct approach for always-on metrics. The profiling bridge (§10) is a supplement, not a primary data source.

2.2 Admin Dashboard Metrics

The MetricsCollector collects server, player, and world metrics with 1-second TTL and broadcasts via WebSocket. Historical data is retained in a 1,440-point circular buffer (24 hours at 1-minute resolution). After this design, the admin UI continues using its MetricsCollector for WebSocket streaming. Prometheus becomes authoritative for alerting, historical queries, and Grafana dashboards.

Unified data source: MetricsCollector is refactored to read from Prometheus Gauge/Counter objects instead of computing values independently. This eliminates duplicate computation and prevents value divergence between what the admin UI shows and what Prometheus exposes. MetricsCollector.get_server_metrics() reads values via the public prometheus_client.REGISTRY.get_sample_value("maid_players_online") API, then formats for the WebSocket protocol. Do not access private Gauge._value.get() — it is an internal implementation detail of prometheus_client and subject to change. The circular buffer remains for admin UI sparkline history, fed from the same Prometheus-authoritative values.

2.3 AI Rate Limiter & Providers

The RateLimiter tracks per-player RPM, daily token usage, and global limits via check_and_reserve(). Each AI provider returns {"prompt_tokens": int, "completion_tokens": int} per request.

Critical gaps: - No counter for request denials vs. approvals - No cost calculation (token counts exist but pricing is not applied) - No per-model aggregation or cumulative tracking - Streaming responses (complete_streaming()) discard usage data entirely in all three providers — all token counting silently stops for streaming calls

2.4 Logging

All ~131 source files use logging.getLogger(__name__) with unstructured text output — no JSON, no correlation IDs, no context propagation.

2.5 Summary of Gaps

Capability Current State Target State
Log format Unstructured text JSON with correlation IDs
Metrics export None (internal only) Prometheus /metrics endpoint
Tracing None OpenTelemetry spans (optional)
Health checks None /healthz, /readyz, /livez
AI cost tracking Token counts per-request (not aggregated) Per-provider/model cost in USD
Alerting None Prometheus alerting rules
Dashboards Admin UI only (WebSocket) Grafana dashboards (importable JSON)
Error tracking None Optional Sentry integration

3. Architecture Overview

3.1 System Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                        MAID Game Engine                                  │
│                                                                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐        │
│  │ Commands  │  │ Systems  │  │ EventBus │  │  AI Providers    │        │
│  │ Registry  │  │ (ECS)    │  │          │  │  (via hooks)     │        │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └───────┬──────────┘        │
│       │              │             │                 │                    │
│  ─────┼──────────────┼─────────────┼─────────────────┼────────────────── │
│       │       Lifecycle Hooks (pre/post)             │                    │
│       ▼              ▼             ▼                 ▼                    │
│  ┌──────────────────────────────────────────────────────────────────┐    │
│  │              ObservabilityRegistry (owned by GameEngine)         │    │
│  │  get_meter() → prometheus-client  [required]                    │    │
│  │  get_tracer() → opentelemetry     [optional, no-op fallback]    │    │
│  │  get_logger() → structlog         [required]                    │    │
│  │  get_cost_tracker() → AICostTracker                              │    │
│  │  safe_observe() → error-safe context manager                    │    │
│  └─────────┬──────────────────┬────────────────────┬───────────────┘    │
│             │                  │                     │                    │
│  ┌──────────┴─────┐  ┌───────┴────────┐  ┌────────┴─────────┐         │
│  │ Internal port  │  │  Game/Admin    │  │   OTLP export    │         │
│  │ :9090          │  │  port :8080    │  │   (optional)     │         │
│  │ /metrics       │  │  game traffic  │  │                  │         │
│  │ /healthz       │  │  admin API     │  │                  │         │
│  │ /readyz /livez │  │                │  │                  │         │
│  └───────┬────────┘  └───────┬────────┘  └────────┬─────────┘         │
└──────────┼───────────────────┼─────────────────────┼────────────────────┘
           ▼                   ▼                     ▼
    ┌──────────────┐    ┌───────────┐         ┌──────────┐
    │  Prometheus  │    │  stdout/  │         │  OTLP/   │
    │  scrape      │    │  Loki/ELK │         │  Jaeger  │
    └──────────────┘    └───────────┘         └──────────┘

Key architectural principle: ObservabilityRegistry is a concrete class owned by GameEngine, not a module-level singleton. AI providers, systems, and content packs receive it via constructor injection. This eliminates import-time coupling and makes testing trivial (inject a no-op registry).

# observability/registry.py
from __future__ import annotations
from typing import Protocol, runtime_checkable

@runtime_checkable
class ObservabilityRegistry(Protocol):
    """Central access point for all observability primitives.

    Owned by GameEngine and injected into subsystems. Never imported
    as a module-level singleton.
    """
    def get_meter(self) -> "PrometheusMeter": ...
    def get_tracer(self) -> "Tracer": ...
    def get_logger(self, name: str) -> "BoundLogger": ...
    def get_cost_tracker(self) -> "AICostTracker": ...
    def safe_observe(self) -> "SafeObserveContext": ...

class DefaultObservabilityRegistry:
    """Production implementation backed by prometheus-client, structlog, OTel."""

    def __init__(
        self,
        settings: "ObservabilitySettings",
        event_bus: "EventBus | None" = None,
    ) -> None:
        self._settings = settings
        self._meter = PrometheusMeter()
        self._tracer = setup_tracer(settings)  # no-op if OTel absent
        self._cost_tracker = AICostTracker(event_bus=event_bus)

    def get_meter(self) -> PrometheusMeter:
        return self._meter

    def get_tracer(self) -> Tracer:
        return self._tracer

    def get_logger(self, name: str) -> BoundLogger:
        return structlog.get_logger(name)

    def get_cost_tracker(self) -> AICostTracker:
        return self._cost_tracker

    def safe_observe(self) -> SafeObserveContext:
        return SafeObserveContext(self._meter)

class NullObservabilityRegistry:
    """No-op implementation for testing. All methods return inert stubs."""
    ...

GameEngine creates the registry at startup and passes it to subsystems:

# In GameEngine.__init__():
self._obs = DefaultObservabilityRegistry(settings.observability, self._event_bus)

# AI providers receive via injection:
provider = AnthropicProvider(config, obs_registry=self._obs)

3.2 Module Structure

packages/maid-engine/src/maid_engine/
└── observability/
    ├── __init__.py              # Public API: setup_observability(), profiles
    ├── registry.py              # ObservabilityRegistry protocol + DefaultObservabilityRegistry
    ├── logging.py               # structlog configuration, processors, stdlib bridge
    ├── metrics.py               # Prometheus metric definitions, scrape cache
    ├── tracing.py               # OpenTelemetry tracer setup (no-op if not installed)
    ├── context.py               # MaidContext frozen dataclass, single ContextVar
    ├── health.py                # Health check endpoints (/healthz, /readyz, /livez)
    ├── internal_server.py       # Dedicated internal port server (metrics + health)
    ├── middleware.py             # ASGI middleware for request tracing + metrics
    ├── ai_metrics.py            # AI cost calculation, token tracking, budget metrics
    ├── hooks.py                 # Pre/post lifecycle hooks for instrumentation
    ├── safe_observe.py          # safe_observe() context manager, error-safe metric ops
    ├── privacy.py               # Player ID anonymization, command argument redaction
    ├── sentry.py                # Optional Sentry integration
    ├── log_sampling.py          # Adaptive log sampling for high-volume events
    └── bridges/
        ├── __init__.py
        └── profiling_bridge.py  # Bridge ProfileManager → Prometheus (supplemental)

3.3 Dependencies

# pyproject.toml additions
dependencies = [
    "structlog>=24.1.0",
    "prometheus-client>=0.21.0",
]

[project.optional-dependencies]
tracing = [
    "opentelemetry-api>=1.25.0",
    "opentelemetry-sdk>=1.25.0",
]
otlp = [
    "opentelemetry-api>=1.25.0",
    "opentelemetry-sdk>=1.25.0",
    "opentelemetry-exporter-otlp-proto-grpc>=1.25.0",
]
sentry = ["sentry-sdk[asyncio]>=2.0.0"]
monitoring = ["psutil>=5.9.0"]  # Recommended for CPU% and VMS metrics

Note on psutil: Listed as an optional monitoring extra. Without it, maid_process_cpu_percent and maid_process_memory_bytes{type="vms"} will not be populated. RSS is available via the stdlib resource module as a fallback.

3.4 Context Propagation

A single frozen dataclass replaces five independent ContextVar objects, ensuring atomic binding and a single .get() call in the structlog processor:

# observability/context.py
from __future__ import annotations
import contextvars
from dataclasses import dataclass
from uuid import uuid4

@dataclass(frozen=True)
class MaidContext:
    """Immutable request context. Bound once at command/connection entry.

    Frozen to prevent partial mutation — all fields are set atomically.
    """
    correlation_id: str = ""
    player_id: str = ""
    session_id: str = ""
    command: str = ""
    tick_number: int = -1

_maid_context_var: contextvars.ContextVar[MaidContext] = contextvars.ContextVar(
    "maid_context", default=MaidContext()
)

def get_context() -> MaidContext:
    """Return the current MaidContext (never None)."""
    return _maid_context_var.get()

def bind_context(**overrides: object) -> contextvars.Token[MaidContext]:
    """Create a new MaidContext by copying the current one with overrides.

    Returns a Token for restoring the previous context (e.g., in finally blocks).
    """
    current = _maid_context_var.get()
    new_ctx = MaidContext(
        correlation_id=str(overrides.get("correlation_id", current.correlation_id)),
        player_id=str(overrides.get("player_id", current.player_id)),
        session_id=str(overrides.get("session_id", current.session_id)),
        command=str(overrides.get("command", current.command)),
        tick_number=int(overrides.get("tick_number", current.tick_number)),
    )
    return _maid_context_var.set(new_ctx)

def new_correlation_id() -> str:
    """Generate a new correlation ID and bind it into the current context."""
    cid = uuid4().hex[:16]
    bind_context(correlation_id=cid)
    return cid

def bind_player_context(player_id: str, session_id: str) -> None:
    bind_context(player_id=player_id, session_id=session_id)

def clear_context() -> None:
    _maid_context_var.set(MaidContext())

The structlog processor performs a single _maid_context_var.get() call and unpacks all fields:

def _add_maid_context(logger, method_name: str, event_dict: dict[str, Any]) -> dict[str, Any]:
    ctx = _maid_context_var.get()
    if ctx.correlation_id:
        event_dict["correlation_id"] = ctx.correlation_id
    if ctx.player_id:
        event_dict["player_id"] = ctx.player_id
    if ctx.session_id:
        event_dict["session_id"] = ctx.session_id
    if ctx.command:
        event_dict["command"] = ctx.command
    return event_dict

3.5 Metric Naming Conventions

All metrics follow Prometheus naming conventions:

  • Prefix: maid_ for all MAID metrics
  • Subsystem namespaces: maid_engine_, maid_ai_, maid_net_, maid_db_, maid_api_
  • Units in name: _seconds, _bytes, _total (counters)
  • Labels: lowercase, snake_case, bounded cardinality
  • ✅ provider="anthropic", model="claude-3-sonnet", protocol="telnet"
  • ✅ event_domain="combat" (bucketed, ~15 values max)
  • ❌ player_id="uuid" (unbounded — never use as Prometheus label)
  • ❌ npc_id="uuid" (unbounded — use application-level tracking)

4. Metrics Design

4.1 Game Metrics

Metric Name Type Labels Description
maid_players_online Gauge — Currently connected players
maid_players_peak Gauge — Peak concurrent players since restart
maid_tick_duration_seconds Histogram — Duration of each game tick
maid_tick_total Counter — Total ticks processed
maid_tick_overruns_total Counter — Ticks exceeding target interval
maid_engine_uptime_seconds Gauge — Time since engine start
maid_engine_tick_rate Gauge — Configured tick rate (ticks/sec)
maid_engine_last_tick_timestamp Gauge — Unix epoch timestamp of last completed tick (set via time.time(), not time.monotonic() — required for Prometheus time() comparison in alerting rules)
maid_commands_total Counter command, content_pack, status Commands executed (status: success/error/not_found/denied)
maid_command_duration_seconds Histogram command, content_pack Command execution duration
maid_entities_total Gauge type Entities by type (npc/item/room/player/other)
maid_entities_count Gauge — Ground-truth total entity count (independent of tagging)
maid_events_total Counter event_domain Events emitted, bucketed by domain (~15 values)
maid_event_handler_errors_total Counter event_domain Event handler errors by domain
maid_rooms_total Gauge — Total rooms in world
maid_systems_tick_duration_seconds Histogram system_name, content_pack Per-system tick duration. Production default: aggregated to pack-level + top-N (see below).
maid_content_packs_loaded Gauge — Loaded content packs
maid_tick_budget_usage_ratio Histogram — Tick duration as fraction of budget: duration / (1/tick_rate). Values >1.0 mean overrun.
maid_info Info version, python_version Static build info. version sourced from importlib.metadata.version("maid-engine") at startup.
maid_npc_count Gauge tier NPC count by autonomy tier (tier: static/scripted/llm_reactive/llm_autonomous).
maid_npc_plans_total Counter tier Autonomous NPC planning decisions executed.
maid_story_signals_total Counter signal_type Story signals emitted by game systems (~10–15 bounded types).

Cardinality bounds: - command: Bounded to registered command names; unrecognized input labeled _unknown. Max ~500. - event_domain: Bucketed via get_event_domain() prefix mapping. Max ~15 values (see §12). - system_name: Bounded to registered ECS systems. Max ~100 (enforced by content pack registration). - type on entities_total: Fixed set of 5 tag values. - tier on maid_npc_count: Fixed set of 4 values: static, scripted, llm_reactive, llm_autonomous. - signal_type on maid_story_signals_total: Bounded by registered signal types (~10–15). Unrecognized → _other.

Per-system tick histogram aggregation (production kill-switch): In production, emitting a histogram per registered system creates linear time-series growth (~100 systems × 3 packs × 9 buckets = 2,700 series). To prevent this, maid_systems_tick_duration_seconds defaults to pack-level aggregation with top-N system detail:

Mode system_name Label Series Count Config Value
pack_only (default) Omitted; only content_pack label ~3 × 9 = 27 MAID_OBSERVABILITY__SYSTEM_TICK_DETAIL=pack_only
top_n Top N slowest systems (by p99) + _other aggregated ~(N+1) × 9 MAID_OBSERVABILITY__SYSTEM_TICK_DETAIL=top_n
all All registered systems ~300 × 9 = 2,700 MAID_OBSERVABILITY__SYSTEM_TICK_DETAIL=all

Default N for top-N mode: 10 (configurable via MAID_OBSERVABILITY__SYSTEM_TICK_TOP_N=10). The top-N set is re-evaluated every gauge export interval (15s). Systems not in the top-N are aggregated under system_name="_other".

class SystemTickAggregator:
    """Aggregates per-system tick histograms based on configured detail level."""
    def __init__(self, mode: str = "pack_only", top_n: int = 10) -> None:
        self._mode = mode
        self._top_n = top_n
        self._system_p99s: dict[str, float] = {}  # system_name → recent p99

    def observe(self, system_name: str, content_pack: str, duration: float) -> None:
        if self._mode == "pack_only":
            self._pack_hist.labels(content_pack=content_pack).observe(duration)
        elif self._mode == "top_n":
            label = system_name if system_name in self._top_set else "_other"
            self._system_hist.labels(system_name=label, content_pack=content_pack).observe(duration)
        else:  # "all"
            self._system_hist.labels(system_name=system_name, content_pack=content_pack).observe(duration)

4.2 AI Metrics

Metric Name Type Labels Description
maid_ai_requests_total Counter provider, model, status, intent LLM API calls (status: success/error/rate_limited/timeout)
maid_ai_tokens_total Counter provider, model, type, intent Tokens consumed (type: prompt/completion)
maid_ai_cost_dollars Counter provider, model, intent Estimated cost in USD
maid_ai_latency_seconds Histogram provider, model, intent End-to-end LLM request latency
maid_ai_rate_limit_hits_total Counter scope Rate limit denials (scope: per_player/global)
maid_ai_budget_remaining Gauge scope Remaining token budget (scope: global/per_player_max — see §5.5 for semantics)
maid_ai_budget_used_today Gauge scope Tokens used today (scope: global/per_player_max — see §5.5 for semantics)
maid_ai_budget_consumed_tokens_total Counter scope Total tokens consumed (enables rate() burn-rate queries)
maid_ai_active_conversations Gauge — Active NPC conversations
maid_ai_conversation_turns_total Counter — Total conversation turns
maid_ai_circuit_breaker_state Gauge provider Circuit breaker state: 0=closed, 1=open, 0.5=half-open
maid_ai_circuit_breaker_trips_total Counter provider Times the circuit breaker tripped to open state

intent label for AI cost attribution: Every AI request must carry a mandatory intent label identifying the feature that triggered the call. This enables per-feature cost breakdowns and targeted budget controls. The intent label has bounded cardinality — only values from the allowlist are accepted; all others map to _other:

_AI_INTENT_ALLOWLIST = frozenset({
    "dialogue",        # NPC conversation (talk/ask/greet commands)
    "quest_gen",       # Quest seed detection and quest generation
    "narrative_desc",  # AI-generated room/item/NPC descriptions
    "combat_narrate",  # AI-narrated combat events
    "npc_autonomy",    # Autonomous NPC planning and decision-making
    "content_gen",     # Builder-triggered content generation
    "moderation",      # Content safety filtering
})

Callers pass intent through AICompletionEvent and the base LLMProvider.complete() kwargs. If omitted, the intent defaults to "dialogue" (the most common use case) and a WARNING is logged once per caller site.

Per-player and per-NPC cost is NOT tracked via Prometheus labels. Using player_id or npc_id creates unbounded cardinality — counters never reset, so every player who ever uses AI creates a time series persisting until Prometheus retention expires. Per-entity cost attribution is handled by the application-level AICostTracker (§5.4).

4.2.1 Quest Generation Metrics

The quest generation pipeline (seed detection → template selection → LLM generation → validation) is monitored with dedicated metrics:

Metric Name Type Labels Description
maid_quest_seeds_detected_total Counter source Quest seeds detected from story signals (source: dialogue/combat/exploration/social)
maid_quests_generated_total Counter outcome, template Quests generated (outcome: success/validation_error/llm_error/timeout; template: bounded by registered templates)
maid_quest_generation_latency_seconds Histogram — End-to-end quest generation latency (uses AI_LATENCY_BUCKETS)
maid_story_signals_total Counter signal_type Story signals emitted by game systems (signal_type: bounded by signal registry, ~10–15 values)

Cardinality bounds: - source on quest seeds: Fixed set of 4 values. - outcome: Fixed set of 4 values. - template: Bounded by registered quest templates in the content pack (typically <20). Enforced via CardinalityGuard. - signal_type: Bounded by registered signal types (~10–15). Unrecognized signals map to _other.

4.3 Database Metrics

Metric Name Type Labels Description
maid_db_query_duration_seconds Histogram collection, operation Database operation duration
maid_db_queries_total Counter collection, operation, status Total queries (status: success/error)
maid_db_slow_queries_total Counter collection, operation Queries exceeding 100ms threshold

Cardinality: collection is bounded by registered schemas, typically <20. operation is a fixed set: find, insert, update, delete, find_one.

4.4 Network Metrics

Metric Name Type Labels Description
maid_net_connections_total Counter protocol Connections accepted (telnet/websocket)
maid_net_connections_active Gauge protocol Active connections
maid_net_bytes_total Counter protocol, direction Bytes transferred (in/out)
maid_net_disconnections_total Counter protocol, reason Disconnections (clean/timeout/error/kicked)
maid_net_messages_total Counter protocol, type Messages by protocol type (type: text/gmcp/oob for websocket; text/telnet_option for telnet). Enables visibility into web client sub-protocol usage.

4.5 Infrastructure & API Metrics

Metric Name Type Labels Description
maid_process_memory_bytes Gauge type Process memory (rss/vms). VMS requires psutil.
maid_process_cpu_percent Gauge — CPU utilization. Requires psutil.
maid_api_requests_total Counter method, path_template, status_code HTTP API requests
maid_api_request_duration_seconds Histogram method, path_template HTTP request duration
maid_observability_scrape_duration_seconds Histogram — Time to render /metrics response (measures background rendering cost)
maid_observability_series_count Gauge — Approximate active time series count (from registry length). Enables storage-budget monitoring.

path_template normalization: The ASGI middleware must extract request.scope["route"].path (the route template, e.g., /admin/entities/{id}) — NOT request.url.path (the concrete path with UUID). Using concrete paths creates unbounded cardinality.

4.6 Histogram Bucket Profiles

Histograms are the primary driver of time-series cardinality. Three bucket profiles allow operators to trade detail for cost:

# observability/metrics.py
from enum import Enum

class HistogramProfile(str, Enum):
    LOW = "low"      # 6-8 buckets — production default
    MEDIUM = "medium" # 10-12 buckets — staging / troubleshooting
    HIGH = "high"     # full resolution — development / benchmarking

TICK_BUCKETS = {
    HistogramProfile.LOW:    (0.005, 0.01, 0.05, 0.1, 0.25, 0.5),
    HistogramProfile.MEDIUM: (0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0),
    HistogramProfile.HIGH:   (0.0005, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.15, 0.25, 0.5, 1.0, 2.0),
}
COMMAND_BUCKETS = {
    HistogramProfile.LOW:    (0.005, 0.01, 0.05, 0.1, 0.5, 1.0),
    HistogramProfile.MEDIUM: (0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0),
    HistogramProfile.HIGH:   (0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0, 10.0),
}
AI_LATENCY_BUCKETS = {
    HistogramProfile.LOW:    (0.5, 1.0, 2.5, 5.0, 10.0, 30.0),
    HistogramProfile.MEDIUM: (0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0),
    HistogramProfile.HIGH:   (0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0),
}
DB_QUERY_BUCKETS = {
    HistogramProfile.LOW:    (0.001, 0.005, 0.01, 0.05, 0.1, 0.5),
    HistogramProfile.MEDIUM: (0.0005, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.5, 1.0),
    HistogramProfile.HIGH:   (0.0005, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.5, 1.0, 5.0),
}
API_REQUEST_BUCKETS = {
    HistogramProfile.LOW:    (0.005, 0.01, 0.05, 0.1, 0.5, 1.0),
    HistogramProfile.MEDIUM: (0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0),
    HistogramProfile.HIGH:   (0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0, 30.0),
}

Configured via MAID_OBSERVABILITY__HISTOGRAM_PROFILE=low|medium|high. Default: low in production, high in development.

4.7 Estimated Time Series Count

Histogram multipliers assume the LOW profile (production default): 6 defined buckets + 1 le="+Inf" + _count + _sum = 9 series per label combination. MEDIUM ≈ 12, HIGH ≈ 15.

Metric Group Approx. Label Combinations Histogram Multiplier (LOW) Time Series
Engine/tick gauges + counters (no labels) 7 metrics × 1 — 7
maid_tick_duration_seconds histogram 1 9 9
maid_tick_budget_usage_ratio histogram 1 9 9
Per-system tick histogram (pack_only default) ~3 packs 9 27
Per-system tick histogram (all mode, if enabled) ~100 systems × 3 packs = 300 9 (2,700)
Commands counter (command × content_pack × status) ~100 × 3 × 4 = 1,200 — 1,200
Command duration histogram ~100 × 3 = 300 9 2,700
Events + handler errors counters ~15 domains × 2 — 30
NPC autonomy (count, plans, story signals) ~4 tiers + ~15 signal types — 25
AI counters (requests, tokens, cost) with intent ~60 × 7 intents ÷ sparse = ~200 — 200
AI latency histogram with intent 3 × 5 × 3 active intents = 45 9 405
AI misc (budget, rate limit, conversations, circuit breaker) ~10 — 10
Quest generation metrics ~20 9 (latency histogram) 30
Database counters (queries + slow) ~10 × 5 × 3 = 150 — 150
Database query histogram ~10 × 5 = 50 9 450
Network (incl. messages by type) 2 protocols × 2 directions + gauges + messages — 24
Process metrics + observability meta ~8 — 8
API counters ~20 × 4 × ~5 = 400 — 400
API request histogram ~20 × 4 = 80 9 720
Static (info, content_packs, entities, rooms) ~10 — 10
Total (LOW profile, pack_only default) ~6,400
Total (LOW profile, all systems) ~9,100
Total (HIGH profile, all systems, worst-case) multiplier → 15 ~15,000

The default pack_only system tick mode reduces the dominant cardinality contributor from ~2,700 series to ~27, yielding a ~25% reduction in total series compared to v3.2. Operators who need per-system detail can enable top_n or all mode (see §4.1).

4.8 /metrics Endpoint, Scrape Cache, and Internal Port

Dedicated internal port (default 9090): The /metrics, /healthz, /readyz, and /livez endpoints are served on a separate internal port, NOT on the game/admin port (8080). This prevents operational data from being exposed to players and avoids accidental auth requirements on scrape endpoints.

# observability/internal_server.py
async def start_internal_server(
    engine: "GameEngine",
    host: str = "127.0.0.1",
    port: int = 9090,
    metrics_token: str = "",
) -> None:
    """Start the internal observability server on a dedicated port.

    Serves /metrics, /healthz, /readyz, /livez.
    Binds to 127.0.0.1 by default — only reachable from localhost/sidecar.

    If metrics_token is set, /metrics requires Bearer token authentication.
    A startup WARNING is emitted if host is 0.0.0.0 and no token is configured.
    """
    if host != "127.0.0.1" and not metrics_token:
        _logger.warning(
            "metrics_port_unauthenticated",
            host=host, port=port,
            msg="Internal metrics port bound to non-loopback address without token. "
                "Set MAID_OBSERVABILITY__METRICS_TOKEN to require authentication.",
        )
    app = _build_internal_app(engine, metrics_token=metrics_token)
    config = uvicorn.Config(app, host=host, port=port, log_level="warning")
    server = uvicorn.Server(config)
    await server.serve()

Configured via MAID_OBSERVABILITY__INTERNAL_PORT=9090, MAID_OBSERVABILITY__INTERNAL_HOST=127.0.0.1, and MAID_OBSERVABILITY__METRICS_TOKEN (see §11.2). When metrics_token is set, the /metrics endpoint requires a Authorization: Bearer <token> header. Health check endpoints (/healthz, /readyz, /livez) are always unauthenticated — they return no sensitive data and must be accessible to orchestrator probes.

Scrape cache with background rendering: The /metrics response is pre-rendered by a background asyncio task, not on the HTTP request path. The handler returns a reference to pre-rendered bytes, ensuring zero serialization work during scrape:

# observability/metrics.py
class ScrapeCache:
    """Background-rendered /metrics response with configurable TTL.

    A background task calls generate_latest() on a timer. The HTTP handler
    only reads the cached bytes — no serialization on the request path.
    """
    def __init__(self, ttl: float = 1.0) -> None:
        self._ttl = ttl
        self._cached: bytes = b""
        self._render_hist = Histogram(
            "maid_observability_scrape_render_seconds",
            "Time to render /metrics exposition format",
        )

    async def _render_loop(self) -> None:
        """Background task: re-render /metrics bytes on TTL interval."""
        while True:
            start = time.perf_counter()
            try:
                self._cached = generate_latest()
            except Exception:
                pass  # Keep serving stale bytes on render failure
            elapsed = time.perf_counter() - start
            self._render_hist.observe(elapsed)
            await asyncio.sleep(self._ttl)

    def get(self) -> bytes:
        """Return pre-rendered bytes. O(1), no serialization."""
        return self._cached

_scrape_cache = ScrapeCache()

async def metrics_endpoint(request: Request) -> Response:
    """Prometheus metrics scrape endpoint (served on internal port).
    Returns pre-rendered bytes — no work on the request path.
    """
    return Response(content=_scrape_cache.get(), media_type=CONTENT_TYPE_LATEST)

The background render loop is started alongside the internal server in start_internal_server(). The TTL (default 1s) is configurable via MAID_OBSERVABILITY__SCRAPE_CACHE_TTL.

Content packs register custom metrics in on_load() with enforced maid_{pack_name}_ prefix. A label allowlist and cardinality cap are enforced at registration time to prevent unbounded growth from third-party packs:

async def on_load(self, engine: GameEngine) -> None:
    meter = engine.obs.get_meter()
    # register_counter validates prefix and applies cardinality guard
    self._combat_rounds = meter.register_pack_counter(
        pack_name="rpg",
        name="maid_rpg_combat_rounds_total",
        description="Combat rounds processed",
        labelnames=["outcome"],
        label_allowlists={"outcome": {"victory", "defeat", "flee", "draw"}},
        cardinality_cap=50,
    )

Content pack metric guardrails: - Metric name must start with maid_{pack_name}_ — rejected at registration otherwise. - Each label must have an explicit allowlist or a cardinality cap (default 50). Values exceeding the cap are mapped to _other. - Maximum 20 metrics per content pack (configurable via MAID_OBSERVABILITY__MAX_PACK_METRICS=20).

4.9 Multi-Process and Multi-Instance Deployments

MAID is a single-process async application by default, so prometheus_client works out of the box. However, operators may run multiple instances for availability or use process managers (e.g., gunicorn with workers for the admin API). This requires explicit configuration:

Single-instance (default): No special configuration. prometheus_client uses in-process CollectorRegistry. This is the expected deployment model.

Multi-worker process model (e.g., gunicorn with multiple workers for the admin API): - Enable prometheus_client multiprocess mode by setting PROMETHEUS_MULTIPROC_DIR to a shared tmpfs directory. - Each worker writes metrics to memory-mapped files; the /metrics endpoint aggregates across all workers. - Histograms and Summaries use multiprocess_mode="all"; Gauges use multiprocess_mode="liveall" to avoid stale values from dead workers.

# observability/metrics.py — multiprocess support
import os
if "PROMETHEUS_MULTIPROC_DIR" in os.environ:
    from prometheus_client import CollectorRegistry, multiprocess
    _registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(_registry)
else:
    from prometheus_client import REGISTRY as _registry

Multiple independent instances (horizontal scaling behind a load balancer): - Each instance exposes its own /metrics on the internal port. - Prometheus scrapes each instance as a separate target with an instance label. - PromQL queries use sum by (instance)(...) for per-instance views or sum(...) for aggregate. - maid_info gauge distinguishes instances via the auto-added instance label. - AI cost counters aggregate correctly across instances via sum(increase(maid_ai_cost_dollars[24h])).

Configured via MAID_OBSERVABILITY__MULTIPROCESS_DIR= (empty = disabled, path = enable multiprocess mode).

5. AI Cost Monitoring

AI cost monitoring is the #1 priority for this observability design. A runaway NPC conversation can silently exhaust API budgets with no operator visibility.

5.1 Token-to-Cost Calculation

Pricing is loaded from an externally configurable source with fallback to embedded defaults.

Config path resolution for pricing.json follows MAID's standard config directory search order: 1. $MAID_CONFIG_DIR/pricing.json (if MAID_CONFIG_DIR env var is set) 2. ./config/pricing.json (relative to working directory) 3. ~/.config/maid/pricing.json (XDG user config) 4. /etc/maid/pricing.json (system-wide)

# observability/ai_metrics.py
"""AI cost calculation with configurable pricing.

Pricing loaded in order of precedence:
1. MAID_AI__PRICING env var (JSON string)
2. pricing.json file in config directory
3. Embedded DEFAULT_PRICING (fallback — may be stale)
"""

# Embedded defaults (USD per 1M tokens)
DEFAULT_PRICING: dict[str, dict[str, tuple[float, float]]] = {
    "anthropic": {
        "claude-sonnet-4-20250514": (3.00, 15.00),
        "claude-opus-4-20250514": (15.00, 75.00),
        "claude-3-5-haiku-20241022": (0.80, 4.00),
        "claude-3-5-sonnet-20241022": (3.00, 15.00),
        "claude-3-haiku-20240307": (0.25, 1.25),
        "claude-3-opus-20240229": (15.00, 75.00),
        "_default": (3.00, 15.00),
    },
    "openai": {
        "gpt-4o": (2.50, 10.00),
        "gpt-4o-mini": (0.15, 0.60),
        "gpt-4-turbo": (10.00, 30.00),
        "gpt-3.5-turbo": (0.50, 1.50),
        "_default": (2.50, 10.00),
    },
    "ollama": {
        # Local models — no API cost. Compute cost is operator's responsibility.
        "_default": (0.0, 0.0),
    },
}

def load_pricing(config_dir: str | None = None) -> dict:
    """Load pricing from env var, file, or embedded defaults.

    Config directory resolution order:
    1. Explicit config_dir parameter
    2. MAID_CONFIG_DIR env var
    3. ./config/ (relative to cwd)
    4. ~/.config/maid/
    5. /etc/maid/
    """
    env_pricing = os.environ.get("MAID_AI__PRICING")
    if env_pricing:
        try:
            return json.loads(env_pricing)
        except json.JSONDecodeError:
            _logger.warning("Invalid JSON in MAID_AI__PRICING, using defaults")

    search_dirs = [d for d in [
        config_dir,
        os.environ.get("MAID_CONFIG_DIR"),
        os.path.join(os.getcwd(), "config"),
        os.path.expanduser("~/.config/maid"),
        "/etc/maid",
    ] if d]
    for candidate_dir in search_dirs:
        pricing_file = Path(candidate_dir) / "pricing.json"
        if pricing_file.exists():
            try:
                with open(pricing_file) as f:
                    return json.load(f)
            except (json.JSONDecodeError, OSError) as e:
                _logger.warning("Failed to load %s: %s", pricing_file, e)
    _logger.info("Using embedded default AI pricing (may be stale)")
    return DEFAULT_PRICING

def calculate_cost(
    provider: str, model: str,
    prompt_tokens: int, completion_tokens: int,
    pricing: dict | None = None,
) -> float:
    """Calculate cost in USD for a single completion request."""
    prices = (pricing or DEFAULT_PRICING).get(provider, {})
    input_price, output_price = prices.get(model, prices.get("_default", (0.0, 0.0)))
    if model not in prices and "_default" in prices:
        # WARNING level so this is visible in production INFO-level logs
        _logger.warning("Unknown model %s/%s, using provider default pricing", provider, model)
    return (prompt_tokens * input_price + completion_tokens * output_price) / 1_000_000

5.1.1 Runtime Pricing Updates

Pricing can be updated at runtime via the admin API without engine restart:

# Admin API endpoint:
# PUT /admin/ai/pricing
# Body: {"anthropic": {"claude-sonnet-4-20250514": [3.00, 15.00], ...}}
#
# Authentication: Requires valid JWT with admin role.
# See Admin API auth standard (Doc 01, §Admin Authentication).
# Non-admin requests receive 403 Forbidden.

async def update_pricing(request: Request) -> Response:
    """Update AI pricing at runtime. Requires admin role. Persists to DocumentStore."""
    new_pricing = await request.json()
    validate_pricing_schema(new_pricing)
    engine.obs.get_cost_tracker().update_pricing(new_pricing)
    # Persisted to the 'config' collection per Doc 01 config schema.
    await engine.document_store.upsert("config", "ai_pricing", new_pricing)
    return Response(status_code=200)

Load order of precedence: 1. DocumentStore config/ai_pricing (runtime updates survive restart — see Doc 01 §Config Collection for schema) 2. MAID_AI__PRICING env var (JSON string) 3. pricing.json file in config directory 4. Embedded DEFAULT_PRICING (fallback — may be stale)

5.2 AI Provider Instrumentation via Lifecycle Hooks

Providers never touch observability code directly. Instead, the base LLMProvider class implements timing and metrics in a wrapper around the abstract _do_complete() method. The safe_observe() context manager ensures metric failures never crash the engine.

# observability/safe_observe.py
import time
from contextlib import contextmanager
from typing import Generator

class SafeObserveContext:
    """Error-safe observability wrapper.

    Catches all exceptions in metric/logging operations, logs at most
    once per 5 minutes per metric name, and increments a meta-counter.
    """
    _last_logged: dict[str, float] = {}  # class-level: metric_name → monotonic timestamp
    LOG_INTERVAL = 300.0  # 5 minutes

    def __init__(self, meter: "PrometheusMeter") -> None:
        self._meter = meter
        self._errors_counter = meter.counter(
            "maid_observability_errors_total",
            "Errors encountered in observability code",
            ["source"],
        )

    @contextmanager
    def __call__(self, source: str = "unknown") -> Generator[None, None, None]:
        try:
            yield
        except Exception as e:
            self._errors_counter.labels(source=source).inc()
            now = time.monotonic()
            last = self._last_logged.get(source, 0.0)
            if now - last > self.LOG_INTERVAL:
                self._last_logged[source] = now
                _logger.warning(
                    "observability_error",
                    source=source,
                    error=str(e),
                    note="suppressed for 5 min",
                )


# ai/providers/base.py — instrumentation via hooks in base class
class LLMProvider(ABC):
    """Base class for all LLM providers.

    Subclasses implement _do_complete() only. Observability is handled
    by the base class wrapper — providers never import metric objects.
    """
    def __init__(self, config: ProviderConfig, obs: ObservabilityRegistry) -> None:
        self._config = config
        self._obs = obs

    async def complete(self, messages: list[Message], **kwargs: Any) -> CompletionResult:
        """Instrumented wrapper — subclasses MUST NOT override this."""
        start = time.perf_counter()
        meter = self._obs.get_meter()
        safe = self._obs.safe_observe()
        intent = kwargs.pop("intent", "dialogue")  # Mandatory intent for cost attribution
        try:
            result = await self._do_complete(messages, **kwargs)
            elapsed = time.perf_counter() - start
            with safe("ai_metrics"):
                labels = {"provider": self.provider_name, "model": self.model_name,
                          "intent": intent}
                meter.counter("maid_ai_requests_total",
                    labels={**labels, "status": "success"}).inc()
                meter.histogram("maid_ai_latency_seconds",
                    labels=labels).observe(elapsed)
                meter.counter("maid_ai_tokens_total",
                    labels={**labels, "type": "prompt"}).inc(result.usage.prompt_tokens)
                meter.counter("maid_ai_tokens_total",
                    labels={**labels, "type": "completion"}).inc(result.usage.completion_tokens)
                cost = calculate_cost(
                    self.provider_name, self.model_name,
                    result.usage.prompt_tokens, result.usage.completion_tokens,
                )
                meter.counter("maid_ai_cost_dollars",
                    labels=labels).inc(cost)
            return result
        except Exception as e:
            with safe("ai_metrics"):
                meter.counter("maid_ai_requests_total",
                    labels={"provider": self.provider_name, "model": self.model_name,
                            "intent": intent, "status": "error"}).inc()
            raise

    @abstractmethod
    async def _do_complete(self, messages: list[Message], **kwargs: Any) -> CompletionResult:
        """Provider-specific completion. No observability code here."""
        ...

Meta-alert for observability degradation:

- alert: MaidObservabilityDegraded
  expr: "rate(maid_observability_errors_total[5m]) > 0"
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Observability subsystem errors detected"
    description: "{{ $labels.source }} is failing. Metrics may be incomplete."

5.3 Streaming Response Fix

Problem: Streaming AI responses (complete_streaming()) return AsyncIterator[str], discarding all usage data. When streaming is used for NPC dialogue, cost tracking silently stops.

Solution: Change the streaming return type to AsyncIterator[CompletionChunk] where the final chunk carries usage data:

@dataclass(frozen=True)
class CompletionChunk:
    """A single chunk from a streaming completion."""
    text: str
    is_final: bool = False
    usage: TokenUsage | None = None  # Populated only on final chunk

@dataclass(frozen=True)
class TokenUsage:
    prompt_tokens: int
    completion_tokens: int

# Base LLMProvider handles streaming instrumentation:
async def complete_streaming(
    self, messages: list[Message], **kwargs: Any,
) -> AsyncIterator[CompletionChunk]:
    """Instrumented streaming wrapper. Yields CompletionChunk objects.

    Guarantees final usage emission even on cancellation (GeneratorExit)
    or consumer disconnect. Uses try/finally to ensure metrics are recorded
    with whatever partial usage data is available.
    """
    start = time.perf_counter()
    safe = self._obs.safe_observe()
    intent = kwargs.pop("intent", "dialogue")
    labels = {"provider": self.provider_name, "model": self.model_name, "intent": intent}
    last_usage: TokenUsage | None = None
    recorded = False
    try:
        async for chunk in self._do_complete_streaming(messages, **kwargs):
            if chunk.usage:
                last_usage = chunk.usage
            yield chunk
            if chunk.is_final and chunk.usage:
                elapsed = time.perf_counter() - start
                with safe("ai_stream_metrics"):
                    self._record_completion_metrics(
                        elapsed, chunk.usage.prompt_tokens,
                        chunk.usage.completion_tokens, status="success",
                        intent=intent,
                    )
                recorded = True
    except GeneratorExit:
        # Consumer cancelled (disconnect, timeout). Record partial usage.
        if not recorded and last_usage:
            elapsed = time.perf_counter() - start
            with safe("ai_stream_metrics"):
                self._record_completion_metrics(
                    elapsed, last_usage.prompt_tokens,
                    last_usage.completion_tokens, status="success",
                    intent=intent,
                )
            recorded = True
        raise
    except Exception:
        with safe("ai_stream_metrics"):
            self._obs.get_meter().counter("maid_ai_requests_total",
                labels={**labels, "status": "error"}).inc()
        recorded = True
        raise
    finally:
        # Fallback: if no usage was ever recorded (e.g., stream produced
        # no final chunk and no exception), record at least the request count.
        if not recorded:
            with safe("ai_stream_metrics"):
                self._obs.get_meter().counter("maid_ai_requests_total",
                    labels={**labels, "status": "success"}).inc()

Per-provider usage extraction: - Anthropic: message_delta event includes usage in final event - OpenAI: stream_options={"include_usage": true} on request, usage in final chunk - Ollama: Final chunk includes eval_count and prompt_eval_count

This is a breaking change to the LLMProvider interface. All three providers must be updated simultaneously. Since MAID is pre-deployment, there are no backwards compatibility concerns.

5.4 Event-Driven Per-Player/Per-NPC Cost Tracking

Per-entity cost is tracked by AICostTracker — a first-class engine component that subscribes to AICompletionEvent via EventBus, not called directly by providers.

@dataclass(frozen=True)
class AICompletionEvent(Event):
    """Emitted by LLMProvider base class after every completion."""
    provider: str
    model: str
    prompt_tokens: int
    completion_tokens: int
    cost_usd: float
    intent: str = "dialogue"  # Feature attribution — see §4.2 for allowlist
    conversation_id: str | None = None
    player_id: str | None = None
    npc_id: str | None = None
    latency_seconds: float = 0.0

@dataclass
class ConversationCostRecord:
    conversation_id: str
    player_id: str
    npc_id: str
    npc_name: str
    provider: str
    model: str
    total_prompt_tokens: int = 0
    total_completion_tokens: int = 0
    total_cost_usd: float = 0.0
    turn_count: int = 0
    last_turn_at: float = 0.0

    def record_turn(self, prompt_tokens: int, completion_tokens: int) -> float:
        cost = calculate_cost(self.provider, self.model, prompt_tokens, completion_tokens)
        self.total_prompt_tokens += prompt_tokens
        self.total_completion_tokens += completion_tokens
        self.total_cost_usd += cost
        self.turn_count += 1
        self.last_turn_at = time.time()
        return cost

class AICostTracker:
    """Event-driven AI cost tracking with per-entity attribution.

    Subscribes to AICompletionEvent via EventBus. Decoupled from providers.
    Provides top-N queries, daily rollups, and conversation-level detail
    without creating unbounded Prometheus time series.

    Also exports aggregated budget gauges to Prometheus on each event,
    replacing the gauge export loop for AI-specific metrics.
    """
    CONVERSATION_TTL_SECONDS: float = 3600.0  # 1 hour since last turn

    def __init__(self, event_bus: "EventBus | None" = None) -> None:
        self._conversations: dict[str, ConversationCostRecord] = {}
        self._daily_rollups: dict[str, DailyCostRollup] = {}
        self._pricing: dict | None = None
        if event_bus is not None:
            event_bus.subscribe(AICompletionEvent, self._on_completion)

    async def _on_completion(self, event: AICompletionEvent) -> None:
        """Handle an AI completion event — update per-entity tracking."""
        if event.conversation_id and event.player_id:
            record = self._conversations.setdefault(
                event.conversation_id,
                ConversationCostRecord(
                    conversation_id=event.conversation_id,
                    player_id=event.player_id,
                    npc_id=event.npc_id or "",
                    npc_name="",
                    provider=event.provider,
                    model=event.model,
                ),
            )
            record.record_turn(event.prompt_tokens, event.completion_tokens)
        self._update_daily_rollup(event)

    def update_pricing(self, pricing: dict) -> None:
        """Update pricing at runtime (called from admin API)."""
        self._pricing = pricing

    def get_top_spenders(self, n: int = 10) -> list[tuple[str, float]]:
        """Top N players by cost today."""
        ...

    def get_top_npcs(self, n: int = 10) -> list[tuple[str, float]]:
        """Top N NPCs by cost today."""
        ...

    def cleanup_stale_conversations(self) -> int:
        """Remove conversations with no activity for TTL period."""
        cutoff = time.time() - self.CONVERSATION_TTL_SECONDS
        stale = [cid for cid, rec in self._conversations.items()
                 if rec.last_turn_at < cutoff and rec.last_turn_at > 0]
        for cid in stale:
            del self._conversations[cid]
        return len(stale)

    def cleanup_old_rollups(self, keep_days: int = 30) -> int:
        """Remove rollups older than keep_days."""
        ...

Decoupling benefits: Providers emit AICompletionEvent as part of the base class instrumentation hook (§5.2). The AICostTracker subscribes via EventBus and never needs to be imported by providers. The tracker can be replaced, extended, or disabled without touching provider code.

5.5 Budget Enforcement Metrics

Budget scope semantics: - scope="global" — The server-wide daily token budget (MAID_AI_DIALOGUE_DAILY_TOKEN_BUDGET). When exhausted, all AI requests are blocked. - scope="per_player_max" — The highest single-player daily usage across all players (MAID_AI_DIALOGUE_PER_PLAYER_DAILY_BUDGET). This is a high-watermark gauge, not per-player series — it shows the worst-case individual consumption without creating unbounded cardinality.

# Updated periodically in gauge export loop:
global_tokens = rate_limiter._global_state.tokens_used_today
ai_budget_used_today.labels(scope="global").set(global_tokens)

if rate_limiter.daily_token_budget > 0:
    remaining = rate_limiter.daily_token_budget - global_tokens
    ai_budget_remaining.labels(scope="global").set(max(0, remaining))

# Per-player high-watermark (for the per_player_max gauge)
max_player_usage = max(
    (p.tokens_used_today for p in rate_limiter._player_states.values()), default=0
)
ai_budget_used_today.labels(scope="per_player_max").set(max_player_usage)

if rate_limiter.per_player_daily_budget > 0:
    remaining_pp = rate_limiter.per_player_daily_budget - max_player_usage
    ai_budget_remaining.labels(scope="per_player_max").set(max(0, remaining_pp))

# Increment cumulative counter for burn-rate queries
ai_budget_consumed_tokens_total.labels(scope="global").inc(delta_since_last)

Per-player AI spend monitoring: The AICostTracker (§5.4) emits an internal warning log when any single player consumes more than a configurable percentage of the global daily budget (default 20%). This enables alerting on abusive or runaway player sessions without Prometheus per-player labels:

# In AICostTracker._on_completion():
if event.player_id:
    player_daily = self._get_player_daily_total(event.player_id)
    if player_daily > self._global_budget * self._player_budget_alert_pct:
        _logger.warning(
            "player_ai_spend_high",
            player_id=event.player_id,  # Anonymized by structlog processor
            spend_pct=round(player_daily / self._global_budget * 100, 1),
            channel="audit",
        )

Configured via MAID_AI__PLAYER_BUDGET_ALERT_PCT=0.20 (default 20%).

5.6 Counter Reset on Restart

Prometheus counters reset to zero on engine restart. For maid_ai_cost_dollars, Prometheus's increase() function handles resets correctly via counter reset detection. For financial reporting, use increase() over the full period, not sum() of raw counter values.

6. Structured Logging

6.1 Log Format

Production (JSON): 

{
  "timestamp": "2025-07-15T14:30:00.123456Z",
  "level": "info",
  "event": "command_executed",
  "logger": "maid_engine.commands.registry",
  "correlation_id": "a1b2c3d4e5f67890",
  "player_id": "p_3f8a1c9b2e4d",
  "command": "look",
  "duration_ms": 2.3
}

Note: player_id shows the HMAC-anonymized pseudonym (p_ prefix), not the raw UUID. See §14.1 for anonymization details. Raw UUIDs never appear in shipped logs when MAID_OBSERVABILITY__ANONYMIZE_PLAYER_IDS=true (default).

Development (console): 

2025-07-15 14:30:00 [info     ] command_executed  correlation_id=a1b2c3d4 command=look duration_ms=2.3

6.2 structlog Configuration

# observability/logging.py
from __future__ import annotations
import logging, sys
from typing import Any
import structlog
from structlog.types import Processor
from maid_engine.observability.context import _maid_context_var

def setup_logging(
    *, json_output: bool = True, log_level: str = "INFO",
    log_sampling_enabled: bool = False,
    anonymize_player_ids: bool = True,
    redact_command_args: bool = True,
) -> None:
    """Configure structlog with MAID context-aware processors."""
    shared_processors: list[Processor] = [
        structlog.contextvars.merge_contextvars,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        _add_maid_context,
        _add_log_channel,
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.UnicodeDecoder(),
    ]
    if log_sampling_enabled:
        shared_processors.insert(0, _sampling_processor)
    if anonymize_player_ids:
        shared_processors.append(_anonymize_player_id)
    if redact_command_args:
        shared_processors.append(_redact_command)

    renderer: Processor = (
        structlog.processors.JSONRenderer() if json_output
        else structlog.dev.ConsoleRenderer(colors=sys.stderr.isatty())
    )

    structlog.configure(
        processors=[*shared_processors, structlog.stdlib.ProcessorFormatter.wrap_for_formatter],
        logger_factory=structlog.stdlib.LoggerFactory(),
        wrapper_class=structlog.stdlib.BoundLogger,
        cache_logger_on_first_use=True,
    )

    # Bridge stdlib logging → structlog processor pipeline.
    # Existing logging.getLogger(__name__) calls gain JSON output + context fields.
    formatter = structlog.stdlib.ProcessorFormatter(
        processors=[structlog.stdlib.ProcessorFormatter.remove_processors_meta, renderer],
        foreign_pre_chain=shared_processors,
    )
    handler = logging.StreamHandler(sys.stdout)
    handler.setFormatter(formatter)
    root_logger = logging.getLogger()
    root_logger.handlers.clear()
    root_logger.addHandler(handler)
    root_logger.setLevel(getattr(logging, log_level.upper(), logging.INFO))

    for noisy in ("asyncio", "websockets", "uvicorn.access"):
        logging.getLogger(noisy).setLevel(logging.WARNING)

def _add_maid_context(logger, method_name: str, event_dict: dict[str, Any]) -> dict[str, Any]:
    """Inject MAID context variables into every log event.
    Single .get() call on one ContextVar — see §3.4.
    """
    ctx = _maid_context_var.get()
    if ctx.correlation_id:
        event_dict["correlation_id"] = ctx.correlation_id
    if ctx.player_id:
        event_dict["player_id"] = ctx.player_id
    if ctx.session_id:
        event_dict["session_id"] = ctx.session_id
    if ctx.command:
        event_dict["command"] = ctx.command
    return event_dict

6.3 Log Level Policy

Level Usage Examples
CRITICAL Unrecoverable errors MemoryError in tick loop, data corruption
ERROR Recoverable errors Handler exception, AI provider failure, storage write failure
WARNING Degraded operation Tick overrun, rate limit approaching, unknown AI model pricing
INFO Significant state changes Engine start/stop, player connect/disconnect, command executed
DEBUG Detailed diagnostics (sampled) Event dispatch, entity creation, query execution

6.4 Migration Strategy

Existing logging.getLogger(__name__) calls continue to work unchanged via the stdlib bridge.

Phase 1: Configure structlog bridge — all ~131 existing log sites gain JSON output and context fields with zero code changes.

Phase 2: Gradually migrate high-value modules to structlog.get_logger():

# Before (works via bridge):
_logger = logging.getLogger(__name__)
_logger.info("Command executed: %s", command_name)

# After (gains bound context):
_logger = structlog.get_logger()
_logger.info("command_executed", command=command_name, duration_ms=elapsed)

Migration priority: core/engine.py → core/events.py → commands/registry.py → ai/providers/*.py → net/session.py → all remaining modules (opportunistic).

Honest assessment: Tiers 1–4 (~10 modules) will be explicitly migrated. The remaining ~120 files migrate when touched for other work. The stdlib bridge ensures all modules produce structured JSON regardless. New code should always use structlog.get_logger().

pytest compatibility: The stdlib bridge routes through logging handlers, so caplog captures output. Structlog-native loggers need structlog.testing.capture_logs(). Document which capture method to use in developer guide.

6.5 Adaptive Log Sampling

# observability/log_sampling.py
@dataclass
class AdaptiveSampler:
    """Adjusts sampling rates based on observed event frequency.

    Error and slow-path events are NEVER sampled — they always pass through.
    Only high-volume operational events (DEBUG/INFO) are subject to adaptive sampling.
    """
    base_rate: int = 1
    high_watermark_per_sec: float = 100.0
    low_watermark_per_sec: float = 10.0
    max_drop_rate: int = 1000
    min_drop_rate: int = 1

    def should_log(self, event_name: str, *, is_error: bool = False, is_slow: bool = False) -> bool:
        # Error and slow-path events always pass through
        if is_error or is_slow:
            return True
        self._event_count += 1
        now = time.monotonic()
        elapsed = now - self._window_start
        if elapsed >= 10.0:
            events_per_sec = self._event_count / elapsed
            self._adjust_rate(events_per_sec)
            self._event_count = 0
            self._window_start = now
        return self._event_count % self._current_rate == 0

# Pre-configured samplers
tick_sampler = AdaptiveSampler(base_rate=100, high_watermark_per_sec=10.0)
websocket_sampler = AdaptiveSampler(base_rate=50, high_watermark_per_sec=100.0)
event_dispatch_sampler = AdaptiveSampler(base_rate=10, high_watermark_per_sec=50.0)

Error/slow-path exceptions from sampling: Events tagged with is_error=True (any WARNING+ log) or is_slow=True (e.g., tick duration > threshold, query > 100ms) bypass all sampling. This ensures that rare-but-important diagnostic events are never dropped, even during high-volume periods.

The same principle applies to trace sampling: the head-based sampler (§7.3) is overridden for error traces — if any span in the trace records an error, the entire trace is force-sampled regardless of the configured sample rate. This is implemented via a ParentBasedTraceIdRatio sampler with an error-override hook:

def _force_sample_on_error(span: Span) -> None:
    """Post-hook: if span has error status, mark trace as force-sampled."""
    if span.status.status_code == StatusCode.ERROR:
        span.set_attribute("sampling.force", True)
Event Source Volume (100 players) Strategy
Tick completion (DEBUG) 4/sec = 345,600/day Sample 1-in-100
Command execution (INFO) ~200/min = 288,000/day Log all (audit trail)
WebSocket messages (DEBUG) ~50/sec = 4.3M/day Sample 1-in-50
Event dispatch (DEBUG) ~20/sec = 1.7M/day Sample 1-in-10

6.6 Error Storm Rate Limiting

Max 10 identical errors per minute. After the threshold, a single "further occurrences suppressed" message is logged and subsequent duplicates are dropped until the window resets.

6.7 Log Volume Management

Split audit vs. operational channels with separate sinks. Two categories of logs with different retention, sampling, and output destinations:

Channel Content Sampling Retention Sink
Audit Player commands, admin actions, auth events, AI cost Never sampled — complete record required Long (90+ days) Dedicated audit log file or stream (/var/log/maid/audit.log)
Operational Tick debug, event dispatch, WS frames, entity lifecycle Sampled at INFO; always at WARNING+ Short (7–14 days) Standard log output (/var/log/maid/maid.log)

Separate sinks at emission: The structlog configuration routes audit and operational events to different handlers at emission time, not just via post-hoc label filtering. This ensures audit logs survive operational log rotation and enables independent shipping pipelines (e.g., audit → long-term S3 archival, operational → short-retention Loki):

# In setup_logging(): two handlers with channel-based filtering
audit_handler = logging.StreamHandler(open("/var/log/maid/audit.log", "a"))
audit_handler.addFilter(_ChannelFilter("audit"))

operational_handler = logging.StreamHandler(sys.stdout)
operational_handler.addFilter(_ChannelFilter("operational"))

root_logger.addHandler(audit_handler)
root_logger.addHandler(operational_handler)

Channels are distinguished by a channel field in every log event, set by a structlog processor:

_AUDIT_EVENTS = frozenset({
    "command_executed", "player_connected", "player_disconnected",
    "admin_action", "auth_login", "auth_failure", "ai_completion",
    "budget_warning", "entity_created", "entity_destroyed",
})

def _add_log_channel(logger, method_name: str, event_dict: dict[str, Any]) -> dict[str, Any]:
    event_name = event_dict.get("event", "")
    event_dict["channel"] = "audit" if event_name in _AUDIT_EVENTS else "operational"
    return event_dict

Operational INFO sampling: In production, operational-channel INFO logs are sampled at 1-in-10 by default. WARNING and above are never sampled. Audit-channel logs are never sampled regardless of level. Configured via MAID_OBSERVABILITY__OPERATIONAL_LOG_SAMPLE_RATE=10.

6.8 Relationship to AuditLogStore

The existing AuditLogStore writes audit records (commands, admin actions, auth events) to DocumentStore for in-game @audit queries. After this design, audit events are dual-written:

  1. Structured logs (audit channel) → shipped to Loki/ELK for operational queries, dashboards, and long-term retention.
  2. AuditLogStore → continues writing to DocumentStore for in-game admin access via @audit commands.

Structured logging does not supersede AuditLogStore. The two serve different access patterns: operators use Grafana/Loki; in-game admins use @audit. Both read from the same source events via the structlog processor pipeline and EventBus respectively.

DocumentStore audit retention (TTL): The audit collection in DocumentStore must have a defined TTL to prevent unbounded storage growth. Default retention: 90 days. Documents older than the TTL are purged by a background cleanup task running in the gauge export loop:

# In metrics_export_loop():
# Purge stale audit records from DocumentStore
await engine.document_store.delete_older_than(
    collection="audit",
    ttl_days=engine.settings.observability.audit_retention_days,  # Default: 90
)

Configured via MAID_OBSERVABILITY__AUDIT_RETENTION_DAYS=90. The structured-log audit channel (shipped to Loki/S3) provides the authoritative long-term archive; the DocumentStore copy is for convenient in-game access only.

6.9 Bounded Async Logging for Hot Paths

Logging I/O on the tick hot path can cause tick stalls if the log sink blocks (e.g., full pipe, slow network sink). To prevent this:

  1. structlog uses a QueueHandler on the root logger: log records are enqueued to a bounded asyncio.Queue (max 10,000 entries) and flushed by a dedicated writer task. If the queue is full, records are dropped and a maid_logs_dropped_total counter increments.
  2. Tick-path logging must not await: all logging calls on the tick path use fire-and-forget semantics via the queue. The writer task handles I/O asynchronously.
  3. Trace export uses BatchSpanProcessor with bounded queue (default 2,048 spans). Overflow drops spans silently — this is acceptable for sampled traces.
# observability/logging.py
_LOG_QUEUE_MAX = 10_000
_log_queue: asyncio.Queue[dict] = asyncio.Queue(maxsize=_LOG_QUEUE_MAX)
_logs_dropped = Counter("maid_logs_dropped_total", "Log records dropped due to queue overflow")

async def _log_writer_loop() -> None:
    """Background task that drains the log queue to the output handler."""
    while True:
        record = await _log_queue.get()
        try:
            _emit_log_record(record)
        except Exception:
            pass  # Never crash the writer

Configured via MAID_OBSERVABILITY__LOG_QUEUE_SIZE=10000.

7. Distributed Tracing

7.1 Architecture

OpenTelemetry tracing is an optional dependency. When not installed, all tracing resolves to no-ops. The primary value is correlating a player command to its downstream effects — this can also be achieved with correlation IDs in structured logs alone.

Trace: "player executes 'attack goblin'"
├── Span: command.execute [command=attack, pack=classic-rpg]
│   ├── Span: event.emit [event_domain=combat]
│   │   ├── Span: system.tick [system=CombatSystem]
│   │   │   ├── Span: db.query [collection=entities, op=find_one]
│   │   │   └── Span: db.query [collection=entities, op=update]
│   │   └── Span: system.tick [system=HealthSystem]
│   └── Span: event.emit [event_domain=combat]
│       └── Span: ai.complete [provider=anthropic, model=claude-3-haiku]

Deferred events: Events queued via emit_sync() are processed later when the original trace context is no longer active. Deferred events start a new trace with a link to the originating trace ID.

7.2 Tracer Setup

# observability/tracing.py
_TRACING_AVAILABLE = False
_tracer: Any = None

try:
    from opentelemetry import trace
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
    _TRACING_AVAILABLE = True
except ImportError:
    pass

def setup_tracing(*, sample_rate: float = 0.1, exporter: str = "none") -> None:
    """Configure OTel tracing. No-op if exporter='none' or OTel not installed."""
    global _tracer
    if not _TRACING_AVAILABLE or exporter == "none":
        return
    # ... TracerProvider setup with BatchSpanProcessor ...
    _tracer = trace.get_tracer("maid-engine")

@contextmanager
def trace_span(name: str, **attributes: Any) -> Generator:
    """Create a span — no-op if tracing not configured."""
    if _tracer is None:
        yield None
        return
    with _tracer.start_as_current_span(name, attributes=attributes) as span:
        try:
            yield span
        except Exception as e:
            span.set_status(trace.StatusCode.ERROR, str(e))
            span.record_exception(e)
            raise

7.3 Sampling Strategy and Overhead Control

Tracing overhead is controlled via a tracing_mode setting that determines both sampling rate and span verbosity:

Mode Spans Created Sample Rate Use Case
minimal Commands + AI calls only 5% Production default — lowest overhead
default Commands + events + AI + storage 10% Staging / troubleshooting
verbose All of the above + per-system tick spans 50% Development / debugging

Configured via MAID_OBSERVABILITY__TRACING_MODE=minimal|default|verbose.

Head-based sampling: The sampling decision is made once at the entry span (command execution or connection accept) and propagated to all child spans. This avoids partial traces where a parent is sampled but children are not.

def _should_sample(mode: str, sample_rate: float) -> bool:
    """Head-based sampling decision at trace entry point."""
    return random.random() < sample_rate
Environment Tracing Mode Sample Rate Default
Development N/A none (disabled) ✅
Staging default 0.1 (10%)
Production minimal 0.05 (5%)

7.4 Log–Trace Correlation

When tracing is enabled, the trace ID is injected into structlog context via an _add_trace_context processor, enabling correlated log–trace queries in Grafana.

8. SLI/SLO Definitions

Service Level Indicators (SLIs) and Service Level Objectives (SLOs) formalize reliability targets and enable error-budget-based alerting.

8.1 SLO Table

SLO Name SLI Definition Objective Error Budget (30d) Measurement Window
Tick Loop Availability 1 - (time_without_ticks / total_time) 99.9% 43.2 min/month 30 days rolling
Command Latency histogram_quantile(0.99, maid_command_duration_seconds) p99 < 500ms N/A (latency SLO) 30 days rolling
Command Success Rate 1 - (rate(commands{status="error"}) / rate(commands)) 99.5% 0.5% error budget 30 days rolling
AI Dialogue Availability 1 - (rate(ai_requests{status=~"error|timeout"}) / rate(ai_requests)) 95.0% 5% error budget 30 days rolling
Health Check Availability probe_success (from blackbox exporter or internal checks) 99.9% 43.2 min/month 30 days rolling

8.2 Error Budget Calculation

# Tick Loop error budget remaining (fraction)
1 - (
  (1 - (sum_over_time(up{job="maid"}[30d]) / count_over_time(up{job="maid"}[30d])))
  / (1 - 0.999)
)

# Command Success Rate error budget burn
(
  sum(increase(maid_commands_total{status="error"}[30d]))
  / sum(increase(maid_commands_total[30d]))
) / 0.005

8.3 SLO Burn-Rate Alerting

Traditional threshold alerts fire too late or too often. Multi-window burn-rate alerts detect SLO violations at different urgencies:

Alert Severity Fast Window Slow Window Burn Rate Action
Page (critical) 5 min 1 hour 14.4× Wake on-call
Ticket (warning) 30 min 6 hours 6× File ticket, fix within shift
Info 6 hours 3 days 1× Observe, no action required

See §9.2 for the complete alerting rules implementing these burn rates.

9. Dashboard & Alerting

9.1 Grafana Dashboards

Two production-ready dashboards are shipped as importable JSON in docs/dashboards/. Both include full grid positions, datasource template variables (DS_PROMETHEUS), threshold markers, and consistent metric names.

Import: Grafana UI → Dashboards → Import → Upload JSON file.

Dashboard 1: MAID — Game Server Health

Panel Type Query Grid Position
Players Online Stat maid_players_online h=4, w=4, x=0, y=0
Uptime Stat maid_engine_uptime_seconds h=4, w=4, x=4, y=0
Tick Rate Stat maid_engine_tick_rate h=4, w=4, x=8, y=0
Tick Overruns (1h) Stat increase(maid_tick_overruns_total[1h]) h=4, w=4, x=12, y=0
Memory (RSS) Stat maid_process_memory_bytes{type="rss"} / 1048576 h=4, w=4, x=16, y=0
CPU % Stat maid_process_cpu_percent h=4, w=4, x=20, y=0
Tick Duration (p95/p99) Time series histogram_quantile(0.95/0.99, ...) h=8, w=12, x=0, y=4
Commands/sec Time series sum(rate(maid_commands_total[5m])) h=8, w=12, x=12, y=4
Entities by Type Time series maid_entities_total by type h=8, w=12, x=0, y=12
Events/sec by Domain Time series topk(10, sum by (event_domain) ...) h=8, w=12, x=12, y=12
Active Connections Time series maid_net_connections_active by protocol h=8, w=12, x=0, y=20
DB Query Latency (p95) Time series histogram_quantile(0.95, ...) by collection h=8, w=12, x=12, y=20
Network Bandwidth Time series rate(maid_net_bytes_total[5m]) h=8, w=24, x=0, y=28

Dashboard 2: MAID — AI Cost Overview

Panel Type Query Grid Position
Daily AI Cost (USD) Stat sum(increase(maid_ai_cost_dollars[24h])) h=4, w=6, x=0, y=0
Budget Remaining (%) Gauge Budget ratio calculation h=4, w=6, x=6, y=0
Active Conversations Stat maid_ai_active_conversations h=4, w=6, x=12, y=0
Conversation Turns Today Stat increase(maid_ai_conversation_turns_total[24h]) h=4, w=6, x=18, y=0
Cost by Provider (24h) Pie chart sum by (provider) (increase(maid_ai_cost_dollars[24h])) h=8, w=12, x=0, y=4
Hourly Burn Rate Time series rate(maid_ai_cost_dollars[1h]) * 3600 h=8, w=12, x=12, y=4
Tokens/sec by Model Time series sum by (model, type) (rate(maid_ai_tokens_total[5m])) h=8, w=12, x=0, y=12
AI Latency p99 Time series histogram_quantile(0.99, ...) by provider h=8, w=12, x=12, y=12
Rate Limit Hits/min Time series rate(maid_ai_rate_limit_hits_total[5m]) * 60 h=8, w=12, x=0, y=20
Requests by Status Time series sum by (status) (rate(maid_ai_requests_total[5m])) h=8, w=12, x=12, y=20
Budget Burn-Down Time series maid_ai_budget_remaining{scope="global"} h=8, w=24, x=0, y=28

Complete dashboard JSON files are provided in docs/dashboards/maid-server-health.json and docs/dashboards/maid-ai-cost-overview.json.

9.2 Prometheus Alerting Rules

Alert rules are shipped as a Jinja2/envsubst template (deploy/prometheus/rules/maid_alerts.yml.j2) with tunable values in a companion maid_alert_config.yml. Operators customize thresholds without editing PromQL:

# deploy/prometheus/maid_alert_config.yml — operator-tunable values
tick_stall_threshold_seconds: 2
memory_critical_bytes: 2147483648      # 2 GB
memory_warning_bytes: 1073741824       # 1 GB
ai_daily_cost_warning_usd: 50
ai_budget_critical_pct: 0.10
command_error_rate_threshold: 0.05
connection_spike_rate: 20
db_slow_query_rate: 1
event_handler_error_rate: 0.5
ai_provider_error_rate: 0.10
ai_latency_p95_threshold: 10
story_signal_drought_minutes: 15       # Deadman switch for quest pipeline
quest_generation_error_rate: 0.50      # Max acceptable quest gen failure rate

# deploy/prometheus/rules/maid_alerts.yml (rendered from template)
groups:
  - name: maid_meta
    rules:
      # Dead-man's switch — fires if MAID target disappears entirely
      - alert: MaidTargetDown
        expr: "up{job=\"maid\"} == 0"
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "MAID target is down — Prometheus cannot scrape"
          runbook_url: "https://docs.maid.dev/runbooks/target-down"

      # Absent metric — fires if maid_tick_total stops existing
      - alert: MaidMetricsAbsent
        expr: "absent(maid_tick_total)"
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "MAID core metrics absent — process may have crashed"
          runbook_url: "https://docs.maid.dev/runbooks/metrics-absent"

      # Observability subsystem degradation
      - alert: MaidObservabilityDegraded
        expr: "rate(maid_observability_errors_total[5m]) > 0"
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Observability errors in {{ $labels.source }}"
          runbook_url: "https://docs.maid.dev/runbooks/observability-degraded"

  - name: maid_slo_burn_rate
    rules:
      # Tick Loop SLO — multi-window burn rate (page)
      - alert: MaidTickLoopSLOPageBurn
        expr: |
          (
            (1 - avg_over_time(maid_tick_loop_healthy[5m])) / (1 - 0.999) > 14.4
          ) and (
            (1 - avg_over_time(maid_tick_loop_healthy[1h])) / (1 - 0.999) > 14.4
          )
        for: 0s
        labels:
          severity: critical
          slo: tick_loop_availability
        annotations:
          summary: "Tick loop SLO burn rate critical — page"
          runbook_url: "https://docs.maid.dev/runbooks/tick-loop-stalled"

      # Tick Loop SLO — multi-window burn rate (ticket)
      - alert: MaidTickLoopSLOTicketBurn
        expr: |
          (
            (1 - avg_over_time(maid_tick_loop_healthy[30m])) / (1 - 0.999) > 6
          ) and (
            (1 - avg_over_time(maid_tick_loop_healthy[6h])) / (1 - 0.999) > 6
          )
        for: 0s
        labels:
          severity: warning
          slo: tick_loop_availability
        annotations:
          summary: "Tick loop SLO burn rate elevated — ticket"
          runbook_url: "https://docs.maid.dev/runbooks/tick-loop-stalled"

      # Command Success Rate SLO — page
      - alert: MaidCommandSLOPageBurn
        expr: |
          (
            sum(rate(maid_commands_total{status="error"}[5m]))
            / sum(rate(maid_commands_total[5m]))
          ) / 0.005 > 14.4
          and
          (
            sum(rate(maid_commands_total{status="error"}[1h]))
            / sum(rate(maid_commands_total[1h]))
          ) / 0.005 > 14.4
        for: 0s
        labels:
          severity: critical
          slo: command_success_rate
        annotations:
          summary: "Command success rate SLO burn rate critical"
          runbook_url: "https://docs.maid.dev/runbooks/command-errors"

  - name: maid_critical
    rules:
      - alert: MaidTickLoopStalled
        expr: |
          (maid_engine_last_tick_timestamp > 0)
          and (time() - maid_engine_last_tick_timestamp > {{ tick_stall_threshold_seconds }})
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "MAID tick loop has stalled"
          description: "No tick completed in {{ $value | humanizeDuration }}."
          runbook_url: "https://docs.maid.dev/runbooks/tick-loop-stalled"

      - alert: MaidAIBudgetExhausted
        expr: 'maid_ai_budget_remaining{scope="global"} <= 0'
        for: 0s
        labels:
          severity: critical
        annotations:
          summary: "AI token budget fully exhausted — NPC dialogue blocked"
          runbook_url: "https://docs.maid.dev/runbooks/ai-budget-exhausted"

      - alert: MaidProcessOOM
        expr: 'maid_process_memory_bytes{type="rss"} > {{ memory_critical_bytes }}'
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "MAID process exceeding 2 GB RSS"
          runbook_url: "https://docs.maid.dev/runbooks/memory-high"

  - name: maid_warning
    rules:
      - alert: MaidTickOverrun
        expr: |
          histogram_quantile(0.95,
            sum(rate(maid_tick_duration_seconds_bucket[5m])) by (le)
          ) > 0.25
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Tick p95 exceeding 250ms target"
          runbook_url: "https://docs.maid.dev/runbooks/tick-overrun"

      - alert: MaidAIBudgetCritical
        expr: |
          maid_ai_budget_remaining{scope="global"} /
          (maid_ai_budget_remaining{scope="global"} +
           maid_ai_budget_used_today{scope="global"}) < {{ ai_budget_critical_pct }}
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "AI budget < 10% remaining"
          runbook_url: "https://docs.maid.dev/runbooks/ai-budget-low"

      - alert: MaidAIHighCostRate
        expr: "sum(rate(maid_ai_cost_dollars[1h])) * 3600 * 24 > {{ ai_daily_cost_warning_usd }}"
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Projected daily AI cost exceeds ${{ ai_daily_cost_warning_usd }}"
          runbook_url: "https://docs.maid.dev/runbooks/ai-cost-high"

      - alert: MaidAIProviderErrors
        expr: |
          sum by (provider) (rate(maid_ai_requests_total{status="error"}[5m]))
          / sum by (provider) (rate(maid_ai_requests_total[5m])) > {{ ai_provider_error_rate }}
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "AI provider {{ $labels.provider }} error rate > 10%"
          runbook_url: "https://docs.maid.dev/runbooks/ai-provider-errors"

      - alert: MaidAIHighLatency
        expr: |
          histogram_quantile(0.95,
            sum by (le, provider) (rate(maid_ai_latency_seconds_bucket[5m]))
          ) > {{ ai_latency_p95_threshold }}
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "AI p95 latency > {{ ai_latency_p95_threshold }}s for {{ $labels.provider }}"
          runbook_url: "https://docs.maid.dev/runbooks/ai-latency-high"

      - alert: MaidCommandErrorRate
        expr: |
          sum(rate(maid_commands_total{status="error"}[5m]))
          / sum(rate(maid_commands_total[5m])) > {{ command_error_rate_threshold }}
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Command error rate > {{ command_error_rate_threshold | float * 100 }}%"
          runbook_url: "https://docs.maid.dev/runbooks/command-errors"

      - alert: MaidConnectionSpike
        expr: "rate(maid_net_connections_total[5m]) > {{ connection_spike_rate }}"
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Connection rate spike: {{ $value | humanize }}/sec"
          runbook_url: "https://docs.maid.dev/runbooks/connection-spike"

      - alert: MaidDBSlowQueries
        expr: "rate(maid_db_slow_queries_total[5m]) > {{ db_slow_query_rate }}"
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Elevated slow query rate"
          runbook_url: "https://docs.maid.dev/runbooks/db-slow-queries"

      - alert: MaidMemoryHigh
        expr: 'maid_process_memory_bytes{type="rss"} > {{ memory_warning_bytes }}'
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "MAID process using > 1 GB RSS"
          runbook_url: "https://docs.maid.dev/runbooks/memory-high"

      - alert: MaidEventHandlerErrors
        expr: "sum(rate(maid_event_handler_errors_total[5m])) > {{ event_handler_error_rate }}"
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Event handler errors > {{ event_handler_error_rate }}/sec"
          runbook_url: "https://docs.maid.dev/runbooks/event-handler-errors"

      - alert: MaidAICircuitBreakerOpen
        expr: "maid_ai_circuit_breaker_state > 0"
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "AI circuit breaker open for {{ $labels.provider }}"
          runbook_url: "https://docs.maid.dev/runbooks/ai-circuit-breaker"

  - name: maid_game_systems
    rules:
      # Story signal drought — deadman switch for narrative pipeline health
      - alert: MaidStorySignalDrought
        expr: |
          (maid_players_online > 5)
          and (increase(maid_story_signals_total[15m]) == 0)
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "No story signals detected for 15 minutes with active players"
          description: "Quest generation pipeline may be stalled. Story signals drive quest seed detection."
          runbook_url: "https://docs.maid.dev/runbooks/story-signal-drought"

      # Quest generation failure rate
      - alert: MaidQuestGenerationErrors
        expr: |
          sum(rate(maid_quests_generated_total{outcome!="success"}[10m]))
          / sum(rate(maid_quests_generated_total[10m])) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Quest generation failure rate > 50%"
          runbook_url: "https://docs.maid.dev/runbooks/quest-generation-errors"

Note: The MaidNoPlayersOnline alert has been intentionally omitted. For most MUD servers, having no players for hours is normal — this alert would be immediately silenced by every operator.

10. Integration with Existing Profiling

10.1 Coexistence Strategy

After implementation, three data sources coexist:

Question Use This Source
"Is the server healthy right now?" Admin UI (1-sec resolution WebSocket)
"Should I be paged?" Prometheus alerting rules
"What happened last night?" Grafana (Prometheus historical data)
"Why was tick #4829 slow?" @timing / @profile (per-system breakdown)

The admin UI continues using MetricsCollector for WebSocket streaming. MetricsCollector reads from Prometheus Gauge/Counter objects (§2.2), so admin UI and Prometheus always show consistent values. Prometheus is authoritative for alerting and Grafana.

10.2 Profiling Bridge (Supplemental)

The ProfilingBridge exports data from active ProfileManager sessions to Prometheus gauges. Because profiling is session-based and disabled by default, the bridge only produces data when an admin runs @profile start. It supplements — not replaces — direct instrumentation.

class ProfilingBridge:
    """Periodically exports ProfileManager data to Prometheus gauges.
    When no profiling sessions are active, the bridge is a no-op."""

    def __init__(self, manager: "ProfileManager", interval: float = 15.0) -> None:
        self._manager = manager
        self._interval = interval

    async def _export_loop(self) -> None:
        while True:
            with self._obs.safe_observe()("profiling_bridge"):
                for session in self._manager.list_sessions():
                    if not session.is_running:
                        continue
                    # Export tick p95/p99, slow query count, tracemalloc allocations
                    # NOTE: MemoryCollector uses tracemalloc (per-allocation tracking),
                    # NOT process RSS. The bridge exports tracemalloc-sourced data as
                    # maid_profiling_tracemalloc_bytes — distinct from the always-on
                    # maid_process_memory_bytes{type="rss"} gauge (§4.5).
            await asyncio.sleep(self._interval)

Admin commands (@profile, @timing, @memory) read directly from collectors and are unaffected.

10.3 Gauge Export Loop

Background task for gauge-style metrics. Also handles AICostTracker cleanup.

async def metrics_export_loop(engine: "GameEngine", interval: float = 15.0) -> None:
    """Periodically update Prometheus gauge metrics."""
    safe = engine._obs.safe_observe()
    while not engine._stop_event.is_set():
        with safe("gauge_export"):
            engine_uptime_seconds.set(engine.uptime)
            engine_tick_rate.set(engine._tick_rate)

            # Entity counts — O(1) via maintained counters (see §10.4)
            for tag, count in engine.world.get_entity_counts().items():
                entities_total.labels(type=tag).set(count)
            entities_count.set(engine.world.total_entity_count)

            # Tick budget utilization
            if engine._last_tick_duration > 0:
                budget = 1.0 / engine._tick_rate
                tick_budget_usage_ratio.observe(engine._last_tick_duration / budget)

            # Process metrics — psutil preferred, resource module fallback
            try:
                import psutil
                proc = psutil.Process()
                mem = proc.memory_info()
                process_memory_bytes.labels(type="rss").set(mem.rss)
                process_memory_bytes.labels(type="vms").set(mem.vms)
                process_cpu_percent.set(proc.cpu_percent(interval=None))
            except ImportError:
                import resource, os
                usage = resource.getrusage(resource.RUSAGE_SELF)
                rss = usage.ru_maxrss
                if os.uname().sysname == "Linux":
                    rss *= 1024
                process_memory_bytes.labels(type="rss").set(rss)

            # AI cost tracker cleanup
            cost_tracker = engine._obs.get_cost_tracker()
            cost_tracker.cleanup_stale_conversations()
            cost_tracker.cleanup_old_rollups()

        await asyncio.sleep(interval)

10.4 O(1) Entity Counting

The gauge export loop previously performed O(n) scans per tag every 15 seconds via query_entities_with_tag(). At 10K entities, this means ~40K inspections per cycle.

Solution: World maintains running counters on create_entity() and destroy_entity():

# core/world.py additions
class World:
    def __init__(self) -> None:
        ...
        self._entity_counts: dict[str, int] = defaultdict(int)
        self._total_entity_count: int = 0

    def create_entity(self, tags: set[str] | None = None) -> EntityID:  # ...
        entity_id = ...  # existing creation logic
        self._total_entity_count += 1
        for tag in (tags or set()):
            self._entity_counts[tag] += 1
        return entity_id

    def destroy_entity(self, entity_id: EntityID) -> None:
        entity = self._entities[entity_id]
        for tag in entity.tags:
            self._entity_counts[tag] = max(0, self._entity_counts[tag] - 1)
        self._total_entity_count -= 1
        ...  # existing destruction logic

    def get_entity_counts(self) -> dict[str, int]:
        """O(1) entity counts by tag. Used by gauge export loop."""
        return dict(self._entity_counts)

    @property
    def total_entity_count(self) -> int:
        return self._total_entity_count

11. Development Mode

11.1 Zero-Config Local Development

In development, observability adds zero overhead by default. Prometheus and tracing are disabled; logging uses colored console output.

# observability/__init__.py
PROFILES: dict[str, dict[str, object]] = {
    "development": {
        "json_logs": False, "log_level": "DEBUG",
        "log_sampling_enabled": False, "metrics_enabled": False,
        "tracing_enabled": False, "profiling_bridge_enabled": False,
    },
    "staging": {
        "json_logs": True, "log_level": "INFO",
        "log_sampling_enabled": True, "metrics_enabled": True,
        "tracing_enabled": True, "tracing_sample_rate": 0.5,
    },
    "production": {
        "json_logs": True, "log_level": "INFO",
        "log_sampling_enabled": True, "metrics_enabled": True,
        "tracing_enabled": True, "tracing_sample_rate": 0.1,
    },
}

Set via MAID_OBSERVABILITY__PROFILE=development|staging|production. Profile values are overridden by explicit env vars (e.g., MAID_OBSERVABILITY__METRICS_ENABLED=true overrides the development profile).

11.2 ObservabilitySettings

class ObservabilitySettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_prefix="MAID_OBSERVABILITY__", env_nested_delimiter="__",
    )

    # Logging
    json_logs: bool = Field(default=True)
    log_level: str = Field(default="INFO")
    log_sampling_enabled: bool = Field(default=True)
    operational_log_sample_rate: int = Field(default=10, description="1-in-N for operational INFO")
    audit_retention_days: int = Field(default=90, description="TTL for DocumentStore audit records")

    # Metrics
    metrics_enabled: bool = Field(default=True)
    internal_host: str = Field(default="127.0.0.1", description="Bind address for internal port")
    internal_port: int = Field(default=9090, description="Dedicated port for /metrics + health")
    metrics_token: str = Field(
        default="",
        description="Bearer token for /metrics endpoint. Required when internal_host != 127.0.0.1",
    )
    gauge_export_interval: float = Field(default=15.0)
    histogram_profile: str = Field(default="low")  # low, medium, high
    scrape_cache_ttl: float = Field(default=1.0, description="Seconds to cache /metrics response")
    system_tick_detail: str = Field(default="pack_only", description="pack_only, top_n, or all")
    system_tick_top_n: int = Field(default=10, description="Number of systems to detail in top_n mode")

    # Cardinality guardrails (operator-tunable per label)
    cardinality_caps: dict[str, int] = Field(
        default={"command": 500, "event_domain": 20, "system_name": 100, "template": 20},
        description="Per-label cardinality caps. Values exceeding cap map to '_other'.",
    )

    # Tracing (requires optional opentelemetry-sdk)
    tracing_enabled: bool = Field(default=False)
    tracing_mode: str = Field(default="minimal")  # minimal, default, verbose
    tracing_sample_rate: float = Field(default=0.05)
    tracing_exporter: str = Field(default="none")  # none, otlp, jaeger, console

    # Sentry (requires optional sentry-sdk)
    sentry_dsn: str = Field(default="")

    # Privacy
    anonymize_player_ids: bool = Field(default=True, description="HMAC player IDs in logs")
    redact_command_args: bool = Field(default=True, description="Redact ALL command args by default")
    command_args_allowlist: set[str] = Field(
        default={"look", "move", "go", "north", "south", "east", "west", "up", "down",
                 "inventory", "score", "who", "help", "quit", "exits"},
        description="Commands whose arguments are safe to log unredacted",
    )

    # Profile
    profile: str = Field(default="production")

Integrates with the existing get_settings() + @lru_cache pattern by adding observability: ObservabilitySettings to the root Settings model.

11.3 Feature Comparison

Feature Development Staging Production
Log format Colored console JSON JSON
Log level DEBUG INFO INFO
Log sampling Disabled Enabled Enabled
Prometheus Disabled Enabled Enabled
Tracing Disabled 50% sample 10% sample
Sentry Disabled Enabled Enabled
CPU overhead ~0% ~0.3% ~0.5%

12. Cardinality Management

12.1 Event Domain Bucketing

The codebase has ~164 unique event classes. Using raw class names as Prometheus labels creates a cardinality explosion. Instead, events are mapped to ~15 bounded domains:

# observability/metrics.py
_EVENT_DOMAIN_MAP: dict[str, str] = {
    "Tick": "engine",      "Startup": "engine",    "Shutdown": "engine",
    "Connect": "network",  "Disconnect": "network",
    "Room": "movement",    "Move": "movement",
    "Combat": "combat",    "Damage": "combat",     "CriticalHit": "combat",
    "Heal": "combat",      "Death": "combat",
    "Item": "inventory",   "Pickup": "inventory",  "Drop": "inventory",
    "Spell": "magic",      "Cast": "magic",
    "Quest": "quest",
    "Guild": "social",     "Faction": "social",
    "Auction": "economy",  "Trade": "economy",
    "Craft": "crafting",
    "NPC": "npc",          "Dialogue": "npc",
    "PvP": "pvp",
    "Weather": "world",    "Time": "world",
}

def get_event_domain(event_type_name: str) -> str:
    """Map event class name to bounded domain label. Max ~15 values."""
    for prefix, domain in _EVENT_DOMAIN_MAP.items():
        if event_type_name.startswith(prefix):
            return domain
    return "other"

12.2 Cardinality Guardrails

Hard caps and allowlists prevent cardinality explosions from unexpected label values. Caps are operator-tunable via MAID_OBSERVABILITY__CARDINALITY_CAPS (see §11.2):

# observability/metrics.py
class CardinalityGuard:
    """Enforces hard caps on label cardinality.

    When a label exceeds its cap, new values are mapped to '_other'.
    Emits a warning log (once per label name) when the cap is hit.
    Caps are operator-tunable per label via ObservabilitySettings.cardinality_caps.
    """
    def __init__(self, caps: dict[str, int] | None = None, default_cap: int = 500) -> None:
        self._caps = caps or {}
        self._default_cap = default_cap
        self._seen: dict[str, set[str]] = defaultdict(set)
        self._warned: set[str] = set()

    def check(self, label_name: str, value: str, allowlist: set[str] | None = None) -> str:
        """Return the value if within cap/allowlist, else '_other'."""
        if allowlist is not None and value not in allowlist:
            return "_other"
        cap = self._caps.get(label_name, self._default_cap)
        seen = self._seen[label_name]
        if value in seen:
            return value
        if len(seen) >= cap:
            if label_name not in self._warned:
                self._warned.add(label_name)
                _logger.warning("cardinality_cap_hit", label=label_name, cap=cap)
            return "_other"
        seen.add(value)
        return value

# Initialized from ObservabilitySettings at startup:
# guard = CardinalityGuard(caps=settings.observability.cardinality_caps)

# Provider/model use explicit allowlists (not cap-based)
_PROVIDER_ALLOWLIST = {"anthropic", "openai", "ollama"}
_STATUS_ALLOWLIST = {"success", "error", "rate_limited", "timeout", "not_found", "denied"}
_AI_INTENT_ALLOWLIST = {"dialogue", "quest_gen", "narrative_desc", "combat_narrate",
                        "npc_autonomy", "content_gen", "moderation"}

12.3 Label Cardinality Bounds Summary

Label Max Values Enforcement Operator-Tunable
event_domain ~15 Prefix mapping with "other" fallback Yes (cardinality_caps.event_domain)
command ~500 Registered commands only; unrecognized → _unknown Yes (cardinality_caps.command)
system_name ~100 (or N+1 in top_n mode) Bounded by content pack registration Yes (cardinality_caps.system_name)
collection <20 Bounded by registered DocumentStore schemas Yes
provider ~5 Fixed allowlist: anthropic, openai, ollama No (allowlist)
model ~15 Known models per provider No (allowlist)
intent 7 Fixed allowlist (see §4.2); unknown → _other No (allowlist)
protocol 2 Fixed: telnet, websocket No
type (net messages) 4 Fixed: text, gmcp, oob, telnet_option No
status / status_code <10 Fixed enum values No
path_template ~30 Route templates, NOT concrete paths Yes
tier (NPC) 4 Fixed: static, scripted, llm_reactive, llm_autonomous No
signal_type ~15 Bounded by signal registry Yes (cardinality_caps.signal_type)
template (quest) <20 Bounded by registered templates Yes (cardinality_caps.template)

12.4 What NOT to Use as Labels

  • player_id, npc_id, session_id — unbounded UUIDs
  • npc_name — arbitrary builder-set strings
  • error_type — any exception class name (unbounded)
  • event_type — raw event class names (~164+ values)
  • Concrete URL paths (/admin/entities/550e8400-...)

13. Health Checks

13.1 Endpoints

Health checks are served on the internal port (default 9090, §4.8), not on the game/admin port.

Endpoint Purpose Kubernetes Probe Response
/healthz Liveness — is the process alive? livenessProbe 200 / 503
/readyz Readiness — can it accept traffic? readinessProbe 200 / 503
/livez Startup — has init completed? startupProbe 200 / 503

13.2 Implementation

Health checks use only verified, public engine APIs. The design requires adding a one-line public property to GameEngine:

Two distinct time bases are used for tick health: - maid_engine_last_tick_timestamp (Prometheus gauge): Set to time.time() (Unix epoch) after each tick. Used by Prometheus alerting rules (time() - maid_engine_last_tick_timestamp), where Prometheus's time() also returns Unix epoch. - last_tick_monotonic (Python property): Returns time.monotonic(). Used by the internal HealthChecker._check_tick_loop() (§13.2) which compares against time.monotonic().

These must not be mixed. The Prometheus alert compares epoch-to-epoch; the health check compares monotonic-to-monotonic.

# Addition to GameEngine (core/engine.py):
@property
def last_tick_monotonic(self) -> float:
    """Monotonic timestamp of the last completed tick.
    Used by internal health checks only — NOT for Prometheus metrics.
    """
    return self._last_tick_time

This avoids accessing private attributes from the observability module, preventing breakage if internal state is refactored.

# observability/health.py
class HealthChecker:
    def __init__(self, engine: "GameEngine") -> None:
        self._engine = engine
        self._startup_complete = False

    def _check_tick_loop(self) -> dict:
        """Verify tick loop ran within 2x expected interval."""
        if not self._engine.is_running:
            return {"status": "fail", "reason": "engine_not_running"}
        last_tick = self._engine.last_tick_monotonic  # Public property
        if last_tick <= 0:
            return {"status": "fail", "reason": "no_ticks_recorded"}
        tick_interval = 1.0 / self._engine.settings.game.tick_rate
        elapsed = time.monotonic() - last_tick
        if elapsed > tick_interval * 2:
            return {"status": "fail", "reason": "tick_stale",
                    "elapsed_seconds": round(elapsed, 3)}
        return {"status": "ok"}

    async def _check_database(self) -> dict:
        """Return cached database status. Never performs I/O on request path."""
        return {"status": "ok" if self._db_healthy else "fail",
                "reason": self._db_fail_reason}

    async def _db_health_loop(self, interval: float = 10.0) -> None:
        """Background loop that checks DB connectivity and caches the result.

        Health check endpoints read the cached status — no I/O per request.
        This prevents a failing DB from making health checks slow/blocking.

        NOTE: DocumentStore.collection_names() is a synchronous API.
        We wrap it in asyncio.to_thread() to avoid blocking the event loop.
        """
        while True:
            try:
                store = self._engine.document_store
                if store is None:
                    self._db_healthy = False
                    self._db_fail_reason = "no_store"
                else:
                    await asyncio.wait_for(
                        asyncio.to_thread(store.collection_names), timeout=5.0,
                    )
                    self._db_healthy = True
                    self._db_fail_reason = ""
            except Exception as e:
                self._db_healthy = False
                self._db_fail_reason = str(e)
            await asyncio.sleep(interval)

13.3 External Health Verification

For bare VM/VPS deployments without Kubernetes:

# Cron-based health check (every minute) — note: internal port
* * * * * curl -sf http://localhost:9090/healthz || \
    echo "MAID down" | mail -s "MAID Alert" admin@example.com

# Telnet smoke test
*/5 * * * * echo "quit" | timeout 5 nc localhost 4000 || echo "Telnet down"

14. Security & Privacy

14.1 Player ID Anonymization in Logs

For GDPR compliance, player UUIDs are anonymized in log output using HMAC with a rotating salt. The original UUID is still available in-process for debugging, but never appears in shipped logs.

# observability/privacy.py
import hashlib
import hmac
import os
import time

class PlayerIDAnonymizer:
    """HMAC-based player ID anonymization for log output.

    Salt rotates daily. Same player maps to same pseudonym within a day
    (enabling log correlation) but different pseudonyms across days.
    """
    def __init__(self, salt_rotation_hours: int = 24) -> None:
        self._rotation_hours = salt_rotation_hours
        self._salt = os.urandom(32)
        self._salt_created = time.monotonic()

    def anonymize(self, player_id: str) -> str:
        if not player_id:
            return ""
        self._maybe_rotate_salt()
        digest = hmac.new(self._salt, player_id.encode(), hashlib.sha256).hexdigest()[:12]
        return f"p_{digest}"

    def _maybe_rotate_salt(self) -> None:
        elapsed = time.monotonic() - self._salt_created
        if elapsed > self._rotation_hours * 3600:
            self._salt = os.urandom(32)
            self._salt_created = time.monotonic()

_anonymizer = PlayerIDAnonymizer()

The structlog processor applies anonymization before log output:

def _anonymize_player_id(logger, method_name: str, event_dict: dict[str, Any]) -> dict[str, Any]:
    if "player_id" in event_dict and event_dict["player_id"]:
        event_dict["player_id"] = _anonymizer.anonymize(event_dict["player_id"])
    return event_dict

Enabled by default in production. Disabled in development via MAID_OBSERVABILITY__ANONYMIZE_PLAYER_IDS=false.

14.2 Command Argument Redaction

Command logs redact ALL arguments by default and only preserve arguments for an explicit whitelist of safe commands. This inverts the previous approach (blacklist of sensitive commands) to prevent PII leaks from any command — including future commands added by content packs.

# Commands whose arguments are safe to log unredacted:
_SAFE_COMMANDS_ALLOWLIST: frozenset[str] = frozenset({
    "look", "move", "go", "north", "south", "east", "west", "up", "down",
    "inventory", "score", "who", "help", "quit", "exits", "commands",
    "areas", "map", "time", "weather",
})

def _redact_command(logger, method_name: str, event_dict: dict[str, Any]) -> dict[str, Any]:
    """Redact command arguments from log events.

    DEFAULT: Redact all arguments. Only commands in the allowlist retain args.
    This prevents PII/secrets leaking through any command, including future
    commands added by content packs.
    """
    cmd = event_dict.get("command", "")
    if cmd:
        parts = cmd.split(maxsplit=1)
        verb = parts[0].lower()
        if verb not in _SAFE_COMMANDS_ALLOWLIST:
            event_dict["command"] = verb  # Verb only, args stripped
        # else: keep full command string (safe to log)
    return event_dict

The allowlist is configurable via MAID_OBSERVABILITY__COMMAND_ARGS_ALLOWLIST (see §11.2). Content packs can register additional safe commands at load time via meter.register_safe_commands({"look", "examine"}).

Enabled by default. Disable with MAID_OBSERVABILITY__REDACT_COMMAND_ARGS=false for development.

14.3 Sentry PII Considerations

When Sentry is enabled, the before_send callback strips player IDs and command arguments from error events. Only the anonymized player pseudonym and command verb are included.

Mandatory HMAC anonymization for set_user(): Sentry's set_user() must receive the HMAC-anonymized player ID (same p_ prefix pseudonym as structured logs, §14.1) — never the raw UUID. This ensures that Sentry's per-user issue grouping works without storing PII:

# observability/sentry.py
def _before_send(event: dict, hint: dict) -> dict | None:
    """Strip PII from Sentry events. Mandate HMAC-anonymized player IDs."""
    ctx = get_context()
    if ctx.player_id:
        sentry_sdk.set_user({"id": _anonymizer.anonymize(ctx.player_id)})
    # Redact command arguments (same allowlist as §14.2)
    if "command" in event.get("extra", {}):
        cmd = event["extra"]["command"]
        verb = cmd.split(maxsplit=1)[0].lower()
        if verb not in _SAFE_COMMANDS_ALLOWLIST:
            event["extra"]["command"] = verb
    return event

Security warning on startup: If sentry_dsn is configured and anonymize_player_ids is false, a WARNING log is emitted at startup: "Sentry enabled without player ID anonymization — raw UUIDs will be sent to Sentry".

14.4 Client-Side Error Ingestion

Web clients may encounter errors (JavaScript exceptions, WebSocket failures, rendering issues) that are invisible to server-side observability. A dedicated endpoint accepts client-side error reports:

POST /api/v1/client-log
Content-Type: application/json
Authorization: Bearer <session_token>

{
  "level": "error",          // error | warn | info
  "message": "WebSocket reconnect failed",
  "source": "ws_handler",    // Bounded: ws_handler | renderer | input | gmcp | unknown
  "user_agent": "...",
  "client_version": "1.2.0",
  "stack_trace": "..."       // Optional, truncated to 4KB server-side
}

Server-side processing: - Rate limited: max 10 reports per player per minute (uses existing RateLimiter infrastructure) - stack_trace truncated to 4KB; no other free-text fields accepted - Player ID anonymized before logging (same HMAC pipeline as §14.1) - Logged as structured log events on the operational channel with event="client_error" - Counted via maid_api_client_errors_total counter with labels: source, level

This endpoint is served on the game/admin port (8080), not the internal port — it must be reachable by web clients.

15. Runbooks & Operational Readiness

15.1 Runbook Template

Every alert references a runbook URL. Runbooks follow this structure:

# Runbook: [Alert Name]

## Summary
One-line description of what this alert means.

## Impact
- What is broken or degraded?
- Who is affected (players, admins, operators)?
- Is data at risk?

## Investigation
1. Step-by-step diagnostic commands
2. Key metrics/logs to check
3. What to look for

## Remediation
1. Immediate mitigation steps
2. Long-term fix if applicable

## Escalation
- When to escalate
- Who to contact
- Relevant Slack/Discord channels

## Verification
- How to confirm the issue is resolved
- What metrics/logs should return to normal

15.2 Critical Alert Runbooks

Shipped as part of Phase 1 for the three highest-severity alerts:

Runbook: MaidTickLoopStalled

Summary: The game engine tick loop has not completed a tick within the expected interval.

Impact: All game simulation is frozen. Players cannot execute commands. NPCs are unresponsive. This is a total outage of gameplay.

Investigation: 1. curl http://localhost:9090/healthz — confirm tick stall vs. full process death 2. Check maid_systems_tick_duration_seconds by system — identify which system is blocking 3. Check process CPU and memory: maid_process_cpu_percent, maid_process_memory_bytes 4. Search logs: jq 'select(.level=="error")' /var/log/maid/maid.log | tail -50 5. Check for deadlocks: kill -SIGQUIT <pid> to dump thread stacks (does not kill process). Do this first — if the engine is truly deadlocked, maid_engine_last_tick_timestamp stops updating and the metric itself becomes stale. A thread dump is the only diagnostic that works in a full deadlock.

Edge case: If the tick loop is deadlocked, the gauge export loop (§10.3) also stops, so maid_engine_last_tick_timestamp freezes. The MaidTickLoopStalled alert will fire, but no further metric updates occur. The thread dump from SIGQUIT is the primary diagnostic tool in this scenario.

Remediation: 1. If a single system is slow: @reload system <system_name> via admin connection 2. If memory is exhausted: restart the engine, file investigation ticket 3. If deadlock suspected: restart the engine, capture thread dump first

Escalation: If not resolved within 15 minutes, escalate to engine maintainer.

Verification: maid_engine_last_tick_timestamp advancing, /healthz returning 200.

Runbook: MaidAIBudgetExhausted

Summary: The global AI token budget is fully consumed. All NPC dialogue is blocked.

Impact: AI-powered NPCs cannot respond. Players interacting with NPCs get error messages. Non-AI gameplay is unaffected.

Investigation: 1. Check maid_ai_budget_used_today{scope="global"} — confirm budget is indeed zero 2. Check rate(maid_ai_cost_dollars[1h]) — was this a sudden spike or gradual burn? 3. Admin API: GET /admin/ai/costs — identify top-spending players/NPCs 4. Check for runaway conversations: high maid_ai_conversation_turns_total rate

Remediation: 1. Increase budget: update MAID_AI_DIALOGUE_DAILY_TOKEN_BUDGET and restart, or use admin API 2. If caused by abuse: identify and ban the player via GET /admin/ai/costs 3. Temporary: reduce MAID_AI_DIALOGUE_PER_PLAYER_RATE_LIMIT_RPM

Escalation: Budget decisions are operator policy. No engineering escalation needed unless the tracker itself is malfunctioning.

Verification: maid_ai_budget_remaining{scope="global"} > 0, NPC dialogue functional.

Runbook: MaidTargetDown

Summary: Prometheus cannot scrape the MAID internal port. The process may have crashed.

Impact: No metrics collection, no alerting for any other condition. Complete monitoring blindness.

Investigation: 1. systemctl status maid or docker ps | grep maid — is the process alive? 2. curl http://localhost:9090/metrics — is the internal port responding? 3. Check system logs: journalctl -u maid --since '5 min ago' 4. Check for OOM kill: dmesg | grep -i oom

Remediation: 1. If process is dead: restart via systemctl restart maid or equivalent 2. If OOM killed: increase memory limits, investigate leak 3. If port conflict: check ss -tlnp | grep 9090

Escalation: If repeated crashes, file bug with crash logs and memory profile.

Verification: up{job="maid"} == 1, /metrics returning data.

15.3 On-Call & Incident Workflow

Severity-to-Response Mapping

Severity Response Time Examples Action
Critical (page) 15 minutes Tick loop stalled, target down, budget exhausted Wake on-call, begin investigation immediately
Warning (ticket) 2 hours (business hours) High error rate, memory warning, AI latency File ticket, fix within shift
Info Next business day SLO burn rate approaching, minor anomalies Review in daily standup

Alertmanager Routing

# deploy/alertmanager/alertmanager.yml (example)
route:
  receiver: default
  group_by: [alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pager
      repeat_interval: 15m
    - match:
        severity: warning
      receiver: ticket
      repeat_interval: 4h
      active_time_intervals:
        - business_hours

receivers:
  - name: pager
    # PagerDuty, Opsgenie, or webhook to Discord/Slack
  - name: ticket
    # Jira, GitHub Issues, or email
  - name: default
    # Slack/Discord channel

time_intervals:
  - name: business_hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '09:00'
            end_time: '18:00'

Single-Operator Mode

Many MUD deployments are run by a single operator. Recommendations:

  • Silence overnight: Use Alertmanager mute_time_intervals for sleeping hours on non-critical alerts
  • Critical only at night: Only page for MaidTargetDown and MaidTickLoopStalled overnight
  • Weekly review: Check error budgets weekly rather than reacting to every info alert
  • Runbook-first: All alerts link to runbooks. Follow the runbook before improvising

16. Testing Strategy

16.1 Unit Tests

Module Key Test Cases
context.py Correlation ID generation, context propagation across async boundaries, isolation between tasks, clear_context()
logging.py JSON output format, console format, context field injection, stdlib bridge captures logging.getLogger() output
metrics.py Counter increment, histogram observe, get_event_domain() bucketing, /metrics returns valid Prometheus text
health.py Liveness uses monotonic elapsed time via public property, readiness calls collection_names(), startup flag transition
ai_metrics.py Cost calculation accuracy, unknown model fallback with WARNING log, zero-cost Ollama, pricing file loading
safe_observe.py Exception suppression, rate-limited logging (1 per 5min), maid_observability_errors_total increments
context.py MaidContext frozen immutability, bind_context() atomicity, isolation across async tasks
privacy.py HMAC determinism within salt window, rotation across windows, empty-string passthrough
registry.py NullObservabilityRegistry returns inert stubs, DefaultObservabilityRegistry wires real backends
tracing.py trace_span() returns no-op when OTel not installed, span creation with attributes when enabled
sentry.py No-op when sentry-sdk absent, _before_send enriches with game context, set_user() for player IDs
log_sampling.py Adaptive rate adjustment, sampling decision correctness

16.2 Integration Tests

Test Verification
Tick instrumentation maid_tick_total increments after World.tick(); engine_last_tick_timestamp updates
Command instrumentation maid_commands_total records correct status with try/except wrapping
AI metrics Token counters increment after complete(); cost calculated correctly
Health endpoints /healthz returns 200 when running; /readyz returns 503 before startup
Event domain bucketing CombatInitiatedEvent → domain "combat"; unknown → "other"
Context propagation Correlation ID appears in structured logs within command execution
Graceful degradation Simulated metric failure does not crash tick loop (via safe_observe)
Rate limiter metrics Denial counter increments on check_and_reserve() rejection
O(1) entity counting World.get_entity_counts() stays consistent through create/destroy cycles
AI event-driven tracking AICompletionEvent → AICostTracker → per-player attribution
Player ID anonymization Production logs contain p_ prefix, not raw UUIDs
Internal port isolation /metrics accessible on 9090, NOT on 8080

16.3 Load Tests

Test Success Criteria
Metric scrape under load (100 players, 15s scrape) /metrics < 50ms p99
Log throughput (10,000 lines/sec for 60s) Zero dropped logs, < 5µs/line
Tick overhead (10,000 ticks with full instrumentation) < 1% avg duration increase
Memory stability (24h with full instrumentation) < 10MB RSS growth
Startup time with observability < 2 seconds added over baseline

17. Performance Impact

17.1 Overhead Estimates

Component CPU Overhead Memory
structlog JSON serialization ~2µs/log line ~1MB pipeline
Prometheus counters/gauges ~0.1µs/operation ~50KB catalog
Prometheus histograms ~0.5µs/observe ~2KB/histogram
/metrics scrape ~5ms per scrape ~100KB temp
OpenTelemetry spans (sampled) ~5µs/span ~1KB/span
OpenTelemetry (not installed) 0 0
Context variables ~0.05µs/get/set ~100B/context
Gauge export loop (15s) <0.1% CPU Negligible

Per-tick aggregate (4 TPS, 20 systems, 100 players): ~60 metric operations × 0.5µs = ~30µs/tick = < 0.02% of 250ms tick budget.

Pre-registered metric handles and PrometheusMeter caching: Hot-path metrics (tick duration, command counters, event counters) must be pre-registered at startup and cached as module-level or instance variables. Do not call meter.counter() or meter.histogram() on every event — this incurs label-set lookup overhead. PrometheusMeter internally caches metric objects by name to avoid re-constructing prometheus_client instruments on repeated calls, but callers should still pre-register labeled children for maximum performance:

# PrometheusMeter caches metric objects internally:
class PrometheusMeter:
    def __init__(self) -> None:
        self._counters: dict[str, Counter] = {}
        self._histograms: dict[str, Histogram] = {}

    def counter(self, name: str, **kwargs: Any) -> Counter:
        if name not in self._counters:
            self._counters[name] = Counter(name, **kwargs)
        return self._counters[name]

# But callers should STILL pre-register labeled children at startup:
self._tick_hist = meter.histogram("maid_tick_duration_seconds").observe
self._cmd_counter_success = meter.counter("maid_commands_total").labels(status="success")
self._cmd_counter_error = meter.counter("maid_commands_total").labels(status="error")

# Hot path — direct call, no lookup:
self._tick_hist(elapsed)
self._cmd_counter_success.inc()

This reduces per-event overhead from ~0.5µs (with label lookup) to ~0.1µs (direct call).

These numbers are estimates, not measurements. The benchmarking plan (§17.3) validates them before production deployment.

17.2 Worst-Case Scenarios

Scenario Mitigation
/metrics scrape during tick processing Scrape cache (§4.8) eliminates re-serialization. Pre-rendered bytes returned.
Log volume spike (error storms) Rate-limited error processor: max 10 identical errors/minute
Trace exporter backpressure BatchSpanProcessor drops spans if queue full (configurable, default 2048)
Gauge export entity counting O(1) via maintained counters (§10.4). No scanning.
psutil.Process().cpu_percent() blocking Called with interval=None (non-blocking); first call returns 0.0, documented behavior
Cardinality explosion from rogue content pack CardinalityGuard (§12.2) maps to _other after cap

17.3 Benchmarking Requirements (Phase 1 Gate)

Before Phase 2 begins, these benchmarks must pass:

  1. Tick overhead: <1% increase over 10,000 ticks
  2. Command latency: <2% increase with 100% trace sampling
  3. Scrape latency: <50ms p99 at /metrics
  4. Scrape contention: <0.1% tick duration increase during scrape
  5. Memory growth: <10MB over 24 hours
  6. Startup time: <2 seconds added

18. Implementation Plan

Phase 1: Foundation (Weeks 1–4) — P0

Week Task Deliverable
1 Structured logging setup observability/logging.py, context.py (MaidContext dataclass); structlog stdlib bridge; all logs gain JSON
1 Configuration ObservabilitySettings with MAID_OBSERVABILITY__* env vars and dev/staging/prod profiles
1 ObservabilityRegistry registry.py with DefaultObservabilityRegistry and NullObservabilityRegistry; GameEngine owns instance
1 Engine API addition Add last_tick_monotonic public property to GameEngine
1 safe_observe context manager safe_observe.py with rate-limited error logging and maid_observability_errors_total counter
1 Privacy processors privacy.py with HMAC player anonymization and command argument redaction
2 Core Prometheus metrics observability/metrics.py with engine, tick, event (domain-bucketed), command metrics; histogram profiles; cardinality guards
2 Internal port server internal_server.py serving /metrics + health on dedicated port (default 9090)
2 Health checks observability/health.py; /healthz, /readyz, /livez with cached DB health via background loop
2 Scrape cache ScrapeCache with 1s TTL for pre-rendered /metrics response
3 Tick loop instrumentation Histogram + counter + safe_observe wrapping in _tick_loop()
3 Command instrumentation Histogram + counter + correlation ID in CommandRegistry.execute()
3 EventBus instrumentation Domain-bucketed event counter in EventBus.emit()
3 O(1) entity counting Maintained counters in World on create/destroy
4 ASGI middleware ObservabilityMiddleware with path_template extraction from route
4 Log sampling + log channels Adaptive sampling, error storm rate limiting, audit/operational split
4 SLO definitions + alerting rules Formal SLI/SLO table, multi-window burn-rate alerts, absent/dead-man alerts
4 Runbook skeletons Ship runbooks for MaidTickLoopStalled, MaidAIBudgetExhausted, MaidTargetDown
4 Alert config template maid_alert_config.yml + Jinja2 template for operator-tunable thresholds
4 Benchmark validation Pass all §17.3 benchmarks before proceeding

Exit criteria: curl localhost:9090/metrics returns Prometheus text. curl localhost:9090/healthz returns 200. All logs are JSON with anonymized player IDs. Benchmarks pass. Runbooks shipped for 3 critical alerts.

Phase 2: AI & Game Metrics (Weeks 5–8) — P0/P1

Week Task Deliverable
5 LLMProvider base class hooks Instrumentation in base complete() wrapper; providers implement _do_complete() only
5 CompletionChunk streaming Change complete_streaming() return type to AsyncIterator[CompletionChunk] with usage in final chunk
5 AI cost calculation ai_metrics.py with configurable pricing (file + env var + DocumentStore)
6 Event-driven AICostTracker Subscribe to AICompletionEvent via EventBus; per-player/per-NPC cost tracking
6 Runtime pricing admin API PUT /admin/ai/pricing with DocumentStore persistence
6 Rate limiter instrumentation Denial counters targeting check_and_reserve()
6 AI circuit breaker metrics maid_ai_circuit_breaker_state gauge, _trips_total counter
7 MetricsCollector unification Refactor MetricsCollector to read from Prometheus Gauge/Counter objects
7 Player/session/world metrics Online gauge, entity gauges (O(1)), room counters, tick budget ratio
7 Database metrics Query histogram and operation counters via wrapper
8 Network metrics Bytes/messages counters, connection gauges
8 Sentry integration Optional error tracking with game context enrichment

Exit criteria: Full metric catalog at /metrics. AI cost tracking functional for both complete() and complete_streaming(). Per-player cost via admin API. MetricsCollector and Prometheus unified.

Phase 3: Tracing, Dashboards & Polish (Weeks 9–14) — P1/P2

Week Task Deliverable
9 OpenTelemetry tracing tracing.py with no-op fallback, tracing_mode support
9–10 Command and AI tracing Root spans for commands, child spans for events/storage/AI
10 Grafana dashboards Complete JSON files with grid positions and template variables
11 Content pack metric API register_counter(), register_histogram() with prefix validation
12 structlog migration (Tier 1–2) Migrate ~10 core modules to structlog.get_logger()
12 Profiling bridge Supplemental export of profiling session data
13 Operator guide Which data source for which question, on-call playbook, Grafana import
13 Alertmanager routing template Example routing config with severity mapping and single-operator guidance
14 Multi-world support (P2) world label for entity/room metrics

Exit criteria: Dashboards importable. Alerts deployed. Tracing functional when OTel installed. Operator guide complete with on-call workflow.

19. Open Questions & Future Work

Open Questions

# Question Status
OQ-1 Should AICostTracker daily rollups persist to DocumentStore? Deferred to post-Phase 3. In-memory only for now with TTL cleanup
OQ-2 Should admin UI eventually read from Prometheus instead of MetricsCollector? Deferred. Unified MetricsCollector (§2.2) reduces urgency
OQ-3 Rate limiter denial categorization relies on string matching — should check_and_reserve() return a machine-readable denial category? Recommended but not required for Phase 2. String matching on "global" is fragile with i18n

Future Work

  • Per-player cost persistence — persist AICostTracker rollups to DocumentStore for survival across restarts
  • Admin UI consolidation — option to read metrics from Prometheus via PromQL API instead of in-process MetricsCollector
  • Model alias normalization — resolve claude-sonnet-4 vs claude-sonnet-4-20250514 in pricing lookup
  • system_name capping — enforce explicit registration-time validation that system names are bounded
  • Prometheus remote write — for operators who want to push metrics rather than scrape

20. Design Decisions Log

# Decision Rationale Alternatives Rejected
D1 structlog for logging Context-variable binding, processor pipeline, stdlib bridge. python-json-logger lacks context binding and requires manual threading. python-json-logger
D2 prometheus-client for metrics Official client, minimal, direct /metrics. No OTLP collector needed. opentelemetry-sdk metrics (heavy, requires collector)
D3 OpenTelemetry as optional MUD is single-process; distributed tracing is over-engineering for most deployments. Correlation IDs in logs provide equivalent visibility. OTel as required dependency
D4 Event domain bucketing ~164 event classes → ~15 domains prevents cardinality explosion. Raw event_type label (unbounded)
D5 Per-entity cost in AICostTracker, not Prometheus player_id/npc_id as Prometheus labels create unbounded cardinality. Application-level tracking provides richer aggregation. Per-player Prometheus counters
D6 MetricsCollector reads from Prometheus objects Eliminates duplicate computation and value divergence between admin UI and Prometheus. Single source of truth. Separate DashboardBridge; independent computation
D7 Public last_tick_monotonic property Health check should not access private _last_tick_time. One-line property prevents coupling to internal state. Accessing _last_tick_time directly
D8 psutil as optional dependency CPU% and VMS require it, but RSS via resource module is sufficient for critical alerts. psutil as required dependency
D9 Remove MaidNoPlayersOnline alert Zero players for hours is normal for most MUDs. Would be immediately silenced. Keep as info-level alert
D10 Remove error_type label from event handler errors Any exception class name creates unbounded cardinality. Use structured logs for error type debugging. Keep error_type label
D11 Profiling bridge as supplement only ProfileManager is session-based, disabled by default. Direct tick loop instrumentation is the primary data source. Bridge as primary data source
D12 Unknown model pricing at WARNING level DEBUG-level warnings are invisible in production. Operators need to know when cost tracking is using fallback pricing. DEBUG-level warning
D13 path_template from route, not URL Concrete URL paths with UUIDs create unbounded cardinality. Route template (/entities/{id}) is bounded. Raw request.url.path
D14 Sentry uses set_user() for player IDs Better cardinality handling than tags. Tag space is limited; set_user() is designed for per-user context. Note: UUID-as-PII depends on operator's privacy policy. Player ID as Sentry tag
D15 Benchmark gate before Phase 2 Overhead claims are estimates. Validation before committing to full implementation prevents discovering performance problems late. Benchmark as follow-up task
D16 maid_entities_count gauge (no labels) Ground-truth total independent of tagging. ECS entities without tags won't appear in maid_entities_total by-type breakdown. Only by-type gauge
D17 ObservabilityRegistry owned by GameEngine, not singleton Dependency injection enables testing with NullObservabilityRegistry. Eliminates import-time coupling. Module-level singletons are untestable. Module-level _provider singleton
D18 safe_observe() replaces bare try/except: pass Rate-limited logging (1 per 5min) prevents silent failures. maid_observability_errors_total enables meta-alerting on degradation. Bare except: pass (silent failures)
D19 Single MaidContext frozen dataclass Five independent ContextVars risk partial-bind bugs. Frozen dataclass ensures atomicity. Single .get() in processor reduces overhead. Five separate ContextVar objects
D20 Dedicated internal port (9090) for /metrics + health Prevents operational data exposure on game/admin port. Avoids auth on scrape endpoints. Standard practice for Kubernetes sidecars. Shared port with auth bypass rules
D21 AICostTracker subscribes via EventBus Decouples from providers. Tracker can be replaced/disabled without touching provider code. Event-driven is consistent with MAID architecture. Direct method calls from providers
D22 CompletionChunk return type for streaming Fixes silent cost tracking failure. Usage in final chunk is supported by all three providers. Breaking change is acceptable pre-deployment. Keep AsyncIterator[str] (broken cost tracking)
D23 HMAC player ID anonymization Rotating salt prevents cross-day correlation while enabling same-day log correlation. GDPR-compatible by default. Raw UUIDs in logs; hash without salt
D24 Multi-window burn-rate SLO alerts Single-threshold alerts are either too noisy or too slow. Burn-rate pattern catches fast burns immediately and slow burns before budget exhaustion. Static threshold alerts
D25 Histogram bucket profiles (low/med/high) Low profile (~6 buckets) reduces time series by ~40% vs full resolution. Operators choose precision vs. cost. Fixed bucket sets
D26 O(1) entity counting via maintained counters O(n) scan of 10K entities every 15s is wasteful. Maintained counters on create/destroy are trivial and O(1). O(n) query_entities_with_tag() every 15s
D27 Mandatory intent label on AI metrics Without feature-level cost attribution, operators cannot distinguish dialogue costs from quest generation or content generation. Bounded allowlist prevents cardinality issues. No intent label (aggregate cost only)
D28 Per-system tick histograms default to pack-level aggregation 100 systems × 9 buckets = 2,700 series is the dominant cardinality driver. Pack-level default with opt-in detail reduces to 27 series while preserving per-pack visibility. Always emit per-system (high cardinality); disable entirely
D29 Whitelist-based command argument redaction Blacklist approach misses future commands that may contain PII. Whitelist (redact ALL by default, allow known-safe) is fail-safe against content pack additions. Blacklist of sensitive commands
D30 Background /metrics rendering (off request path) Eliminates serialization latency from scrape requests. Background task pre-renders bytes on TTL, handler returns cached reference. Lazy cache (serialize on first scrape after TTL expiry)
D31 MAID_OBSERVABILITY__METRICS_TOKEN for non-loopback binding Loopback binding is safe, but operators binding to 0.0.0.0 need authentication. Token-based auth is simpler than TLS client certs for a metrics endpoint. Always require auth; never require auth
D32 Streaming try/finally with GeneratorExit handling Consumer disconnect during streaming silently loses cost data. Explicit GeneratorExit handling with last_usage tracking ensures partial data is always recorded. Only record on successful final chunk
D33 Sentry set_user() with HMAC-anonymized ID Raw UUIDs in Sentry constitute PII. HMAC preserves per-user issue grouping while anonymizing the identifier. Raw UUID; no user context
D34 Operator-tunable cardinality caps Fixed caps may be too restrictive for large deployments or too loose for small ones. Per-label tuning via cardinality_caps dict allows operators to match their Prometheus capacity. Fixed hardcoded caps

Appendix A: Configuration Reference

# Logging
MAID_OBSERVABILITY__JSON_LOGS=true           # JSON (prod) or console (dev)
MAID_OBSERVABILITY__LOG_LEVEL=INFO
MAID_OBSERVABILITY__LOG_SAMPLING_ENABLED=true
MAID_OBSERVABILITY__OPERATIONAL_LOG_SAMPLE_RATE=10  # 1-in-N for operational INFO
MAID_OBSERVABILITY__AUDIT_RETENTION_DAYS=90  # TTL for DocumentStore audit records

# Metrics
MAID_OBSERVABILITY__METRICS_ENABLED=true
MAID_OBSERVABILITY__INTERNAL_HOST=127.0.0.1  # Bind address for internal port
MAID_OBSERVABILITY__INTERNAL_PORT=9090       # Dedicated port for /metrics + health
MAID_OBSERVABILITY__METRICS_TOKEN=           # Bearer token for /metrics (required if host != 127.0.0.1)
MAID_OBSERVABILITY__GAUGE_EXPORT_INTERVAL=15
MAID_OBSERVABILITY__HISTOGRAM_PROFILE=low    # low, medium, high
MAID_OBSERVABILITY__SCRAPE_CACHE_TTL=1.0     # Seconds between background /metrics renders
MAID_OBSERVABILITY__SYSTEM_TICK_DETAIL=pack_only  # pack_only, top_n, all
MAID_OBSERVABILITY__SYSTEM_TICK_TOP_N=10     # Systems to detail in top_n mode

# Cardinality guardrails (JSON dict of label_name → max_values)
MAID_OBSERVABILITY__CARDINALITY_CAPS='{"command":500,"event_domain":20,"system_name":100,"template":20}'

# Tracing (requires optional opentelemetry-sdk)
MAID_OBSERVABILITY__TRACING_ENABLED=false
MAID_OBSERVABILITY__TRACING_MODE=minimal     # minimal, default, verbose
MAID_OBSERVABILITY__TRACING_EXPORTER=none    # none, otlp, jaeger, console
MAID_OBSERVABILITY__TRACING_SAMPLE_RATE=0.05

# Sentry (requires optional sentry-sdk)
MAID_OBSERVABILITY__SENTRY_DSN=

# Privacy
MAID_OBSERVABILITY__ANONYMIZE_PLAYER_IDS=true
MAID_OBSERVABILITY__REDACT_COMMAND_ARGS=true
MAID_OBSERVABILITY__COMMAND_ARGS_ALLOWLIST='["look","move","go","inventory","score","who","help","quit","exits"]'

# AI Cost
MAID_AI__PRICING='{"anthropic":{"claude-sonnet-4-20250514":[3.0,15.0]}}'
MAID_AI__PLAYER_BUDGET_ALERT_PCT=0.20        # Alert when single player uses >20% global budget

# Profile selection
MAID_OBSERVABILITY__PROFILE=production       # development, staging, production

Appendix B: Prometheus Scrape Configuration

# prometheus.yml
scrape_configs:
  - job_name: maid
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets: ["maid-server:9090"]  # Internal port, NOT game port
    metrics_path: /metrics
    # If MAID_OBSERVABILITY__METRICS_TOKEN is set, add:
    # authorization:
    #   type: Bearer
    #   credentials: "<your-metrics-token>"

Appendix C: Log Shipping

Promtail → Loki

scrape_configs:
  # Operational logs (short retention)
  - job_name: maid-operational
    static_configs:
      - targets: [localhost]
        labels:
          job: maid
          channel: operational
          __path__: /var/log/maid/maid.log
    pipeline_stages:
      - json:
          expressions:
            level: level
            correlation_id: correlation_id
            channel: channel
      - labels:
          level:
          channel:
      - timestamp:
          source: timestamp
          format: "2006-01-02T15:04:05.000000Z"

  # Audit logs (long retention — separate Loki tenant or label)
  - job_name: maid-audit
    static_configs:
      - targets: [localhost]
        labels:
          job: maid
          channel: audit
          __path__: /var/log/maid/audit.log
    pipeline_stages:
      - json:
          expressions:
            level: level
            correlation_id: correlation_id
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: "2006-01-02T15:04:05.000000Z"

Filebeat → Elasticsearch

filebeat.inputs:
  - type: log
    paths: ["/var/log/maid/*.log"]
    json.keys_under_root: true
    json.message_key: event

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "maid-logs-%{+yyyy.MM.dd}"

Appendix D: Storage and Retention Sizing

Concrete sizing estimates for a typical deployment (100 concurrent players, LOW histogram profile, pack_only system tick mode, 15s scrape interval):

Backend Data Source Ingestion Rate Daily Volume 30-Day Volume Notes
Prometheus /metrics scrape ~6,400 series × 2 bytes/sample × (86,400s / 15s) = ~75 MB/day ~75 MB/day ~2.2 GB Compressed TSDB; actual disk ~40% of raw with default compression. Reduced from v3.2 by pack_only default.
Loki (operational) Operational logs (JSON) ~500 bytes/line × ~400K lines/day (with sampling) ~200 MB/day ~2.8 GB 14-day retention. With operational sampling at 1-in-10.
Loki (audit) Audit logs (JSON) ~500 bytes/line × ~100K lines/day (never sampled) ~50 MB/day ~4.5 GB 90-day retention. Never sampled, complete record.
Jaeger OTLP traces (5% sample) ~1 KB/span × ~50K spans/day ~50 MB/day ~350 MB 7-day retention. At 5% sampling rate; 50% staging rate → ~500 MB/day
DocumentStore Audit records (in-game @audit) ~200 bytes/record × ~100K records/day ~20 MB/day ~1.8 GB 90-day TTL (configurable via MAID_OBSERVABILITY__AUDIT_RETENTION_DAYS). Background cleanup in gauge export loop.

Total storage (production, 30 days): ~11.7 GB across all backends. Modest by modern standards.

Retention recommendations: - Prometheus: 30 days local, 90+ days via remote write to object storage (Thanos/Cortex) if needed - Loki operational: 14 days - Loki audit: 90 days (separate tenant or label for independent retention) - Jaeger: 7 days (traces are diagnostic, not archival) - DocumentStore audit: 90 days (configurable, §6.8)

Appendix E: Risks and Mitigations

# Risk Likelihood Impact Mitigation
R1 Event type cardinality explosion Resolved High Domain bucketing to ~15 labels (§12)
R2 structlog migration never completes High Low stdlib bridge ensures JSON output regardless
R3 AI pricing table becomes stale High Medium External pricing.json + env var; embedded defaults are fallback only
R4 Per-player labels create unbounded Prometheus series Resolved High AICostTracker application-level tracking (§5.4)
R5 Observability failures crash engine Resolved Critical safe_observe() context manager with rate-limited logging and meta-counter (§5.2)
R6 Streaming AI responses lose cost data Resolved High CompletionChunk with cancellation-safe try/finally usage emission (§5.3)
R7 /metrics exposed without auth Resolved Medium Dedicated internal port (default 9090, bind 127.0.0.1) + optional MAID_OBSERVABILITY__METRICS_TOKEN with startup warning on 0.0.0.0 binding (§4.8)
R8 Health check breaks on engine refactor Resolved High Public last_tick_monotonic property (§13.2, D7)
R9 AICostTracker memory leak from orphaned conversations Mitigated Low TTL-based cleanup in gauge export loop + event-driven architecture reduces orphan risk
R10 Three overlapping metric sources confuse operators Resolved Medium MetricsCollector unified to read from Prometheus objects (§2.2). "Which Data Source" table in operator guide (§10.1)
R11 Per-system tick histograms cause linear series growth Resolved Medium Default pack_only mode with operator opt-in to top_n/all (§4.1)
R12 AI cost unattributable to features Resolved High Mandatory intent label on all AI metrics with bounded allowlist (§4.2)
R13 Command argument PII leak in logs/Sentry Resolved High Whitelist-based redaction: ALL args redacted by default, only safe commands pass through (§14.2)
R14 Quest generation pipeline silent failure Resolved Medium Dedicated quest metrics (§4.2.1) + story signal drought deadman alert (§9.2)
R15 DocumentStore audit collection unbounded growth Resolved Medium Configurable TTL with background cleanup (§6.8), default 90 days
R16 /metrics serialization blocking scrape Resolved Low Background asyncio task pre-renders bytes; handler returns cached reference (§4.8)