Operational Observability — Implementation Plan¶

Design Document: docs/designs/v3.1/06-operational-observability.md Priority: P0 — Critical Path Estimated Duration: 11 weeks across 3 phases

Summary¶

This plan implements the Operational Observability layer for MAID, providing production-grade monitoring, metrics, structured logging, health checks, tracing, and alerting. The implementation adds:

ObservabilityRegistry — protocol-based registry owned by GameEngine (not a singleton) with DefaultObservabilityRegistry and NullObservabilityRegistry implementations
MaidContext — frozen dataclass propagated via ContextVar carrying correlation ID, player ID, session ID, command, and tick number
structlog integration — JSON/console structured logging with stdlib bridge, context processors, and privacy-aware field redaction
Prometheus metrics — PrometheusMeter with caching, CardinalityGuard, histogram profiles, SystemTickAggregator, and ScrapeCache
Health checks — HealthChecker serving /healthz, /readyz, /livez on a dedicated internal port (default 9090)
safe_observe() — context manager wrapping all instrumentation calls with rate-limited error logging and a meta-counter
Privacy — HMAC-based PlayerIDAnonymizer and whitelist-based command argument redaction
Log sampling — AdaptiveSampler with error/slow-path bypass and audit/operational channel split
AI cost tracking — CompletionChunk streaming fix, LLMProvider instrumentation wrapper, event-driven AICostTracker with configurable pricing
OpenTelemetry tracing — optional tracing with no-op fallback and configurable tracing_mode (minimal/default/verbose)
Grafana dashboards — Server Health and AI Cost Overview JSON dashboard definitions
Content pack metric API — register_pack_counter()/register_pack_histogram() with prefix validation and label allowlists
SLO/SLI definitions — multi-window burn-rate alerting with Jinja2 alert rule templates
Runbooks — operational runbooks for MaidTickLoopStalled, MaidAIBudgetExhausted, MaidTargetDown

Existing infrastructure leveraged: - GameEngine at packages/maid-engine/src/maid_engine/core/engine.py with __init__(), start(), stop(), _tick_loop() - EventBus at packages/maid-engine/src/maid_engine/core/events.py with subscribe(), emit(), emit_sync() - World at packages/maid-engine/src/maid_engine/core/world.py with create_entity(), destroy_entity(), tick(delta) - Settings at packages/maid-engine/src/maid_engine/config/settings.py with get_settings(), clear_settings_cache() - LLMProvider at packages/maid-engine/src/maid_engine/ai/providers/base.py with complete(), complete_streaming() - RateLimiter at packages/maid-engine/src/maid_engine/ai/rate_limiter.py with check_and_reserve() - CircuitBreaker at packages/maid-engine/src/maid_engine/ai/circuit_breaker.py with state property, CircuitState enum - LayeredCommandRegistry at packages/maid-engine/src/maid_engine/commands/registry.py with execute() - ProfileManager at packages/maid-engine/src/maid_engine/profiling/manager.py with start_session(), stop_session() - MetricsCollector at packages/maid-engine/src/maid_engine/api/admin/dashboard.py with collect_server_metrics(), collect_player_metrics(), collect_world_metrics() - AuditLogger at packages/maid-engine/src/maid_engine/logging/audit.py - WebServer at packages/maid-engine/src/maid_engine/net/web/server.py - DocumentStore at packages/maid-engine/src/maid_engine/storage/document_store.py

Phase 1: Foundation (Weeks 1–4) — P0¶

1.1 Package Structure and Dependencies¶

Package: maid-engine | Priority: P0 | Dependencies: none

[ ] Create packages/maid-engine/src/maid_engine/observability/__init__.py
[ ] Module docstring explaining the observability framework
[ ] Re-export key public types: ObservabilityRegistry, MaidContext, safe_observe, HealthChecker
[ ] Create empty module files for the observability package:
[ ] packages/maid-engine/src/maid_engine/observability/registry.py
[ ] packages/maid-engine/src/maid_engine/observability/logging.py
[ ] packages/maid-engine/src/maid_engine/observability/metrics.py
[ ] packages/maid-engine/src/maid_engine/observability/tracing.py
[ ] packages/maid-engine/src/maid_engine/observability/context.py
[ ] packages/maid-engine/src/maid_engine/observability/health.py
[ ] packages/maid-engine/src/maid_engine/observability/internal_server.py
[ ] packages/maid-engine/src/maid_engine/observability/middleware.py
[ ] packages/maid-engine/src/maid_engine/observability/ai_metrics.py
[ ] packages/maid-engine/src/maid_engine/observability/hooks.py
[ ] packages/maid-engine/src/maid_engine/observability/safe_observe.py
[ ] packages/maid-engine/src/maid_engine/observability/privacy.py
[ ] packages/maid-engine/src/maid_engine/observability/sentry.py
[ ] packages/maid-engine/src/maid_engine/observability/log_sampling.py
[ ] packages/maid-engine/src/maid_engine/observability/bridges/__init__.py
[ ] Add required dependencies to packages/maid-engine/pyproject.toml:
[ ] structlog>=24.1.0 (required)
[ ] prometheus-client>=0.21.0 (required)
[ ] Add optional dependency extras to packages/maid-engine/pyproject.toml:
[ ] tracing = ["opentelemetry-api>=1.25.0", "opentelemetry-sdk>=1.25.0"]
[ ] otlp = ["opentelemetry-exporter-otlp-proto-grpc>=1.25.0"]
[ ] sentry = ["sentry-sdk[asyncio]>=2.0.0"]
[ ] monitoring = ["psutil>=5.9.0"]
[ ] Run uv sync to verify dependency resolution
[ ] Create test directory packages/maid-engine/tests/observability/ with __init__.py

1.2 ObservabilitySettings Configuration¶

Package: maid-engine | Priority: P0 | Dependencies: 1.1

[ ] Add ObservabilitySettings(BaseSettings) Pydantic model to packages/maid-engine/src/maid_engine/config/settings.py:
[ ] model_config = SettingsConfigDict(env_prefix="MAID_OBSERVABILITY__", env_nested_delimiter="__")
[ ] General:
- [ ] enabled: bool = Field(default=True) — master switch for all observability
[ ] Logging fields:
- [ ] json_logs: bool = Field(default=True) — JSON (prod) or console (dev) output
- [ ] log_level: str = Field(default="INFO")
- [ ] log_sampling_enabled: bool = Field(default=True)
- [ ] operational_log_sample_rate: int = Field(default=10) — 1-in-N for operational INFO
- [ ] audit_retention_days: int = Field(default=90) — TTL for DocumentStore audit records
- [ ] log_queue_size: int = Field(default=10_000) — bounded async log queue
[ ] Metrics fields:
- [ ] metrics_enabled: bool = Field(default=True)
- [ ] internal_host: str = Field(default="127.0.0.1") — bind address for internal port
- [ ] internal_port: int = Field(default=9090) — dedicated port for /metrics + health
- [ ] metrics_token: str = Field(default="") — Bearer token for /metrics (required when host != 127.0.0.1)
- [ ] gauge_export_interval: float = Field(default=15.0)
- [ ] histogram_profile: str = Field(default="compact") — compact (6 buckets) or detailed (10-12 buckets)
- [ ] scrape_cache_ttl: float = Field(default=1.0) — seconds between background /metrics renders
- [ ] system_tick_detail: str = Field(default="pack_only") — pack_only or all
- [ ] max_pack_metrics: int = Field(default=20) — maximum metrics per content pack
[ ] Cardinality guardrails:
- [ ] cardinality_caps: dict[str, int] = Field(default={"command": 500, "event_domain": 20, "system_name": 100, "template": 20})
[ ] Tracing fields (requires optional opentelemetry-sdk):
- [ ] tracing_enabled: bool = Field(default=False)
- [ ] tracing_mode: str = Field(default="minimal") — minimal, default, verbose
- [ ] tracing_sample_rate: float = Field(default=0.05)
- [ ] tracing_exporter: str = Field(default="none") — none, otlp, jaeger, console
[ ] Sentry (requires optional sentry-sdk):
- [ ] sentry_dsn: str = Field(default="")
[ ] Privacy fields:
- [ ] anonymize_player_ids: bool = Field(default=True) — HMAC player IDs in logs
- [ ] redact_command_args: bool = Field(default=True) — redact ALL command args by default
- [ ] command_args_allowlist: set[str] = Field(default={"look", "move", "go", "north", "south", "east", "west", "up", "down", "inventory", "score", "who", "help", "quit", "exits"}) — commands whose args are safe to log
[ ] Profile selection:
- [ ] profile: str = Field(default="production") — development, staging, production
[ ] Define PROFILES dict with preset overrides for development, staging, production (§11.1)
[ ] Environment variable prefix: MAID_OBSERVABILITY__
[ ] Add observability: ObservabilitySettings = ObservabilitySettings() field to Settings class
[ ] Write unit tests in packages/maid-engine/tests/observability/test_settings.py:
[ ] Test default settings
[ ] Test profile overrides (dev/staging/prod)
[ ] Test environment variable binding with MAID_OBSERVABILITY__ prefix

1.3 MaidContext and ContextVar Propagation¶

Package: maid-engine | Priority: P0 | Dependencies: 1.1

[ ] Create MaidContext frozen dataclass in packages/maid-engine/src/maid_engine/observability/context.py:
[ ] @dataclass(frozen=True) — frozen to prevent partial mutation; all fields set atomically
[ ] correlation_id: str = "" — hex UUID4 prefix (16 chars), generated per request/command
[ ] player_id: str = "" — raw player UUID (anonymized only in log output, not in ContextVar)
[ ] session_id: str = ""
[ ] command: str = ""
[ ] tick_number: int = -1
[ ] Define module-level _maid_context_var: ContextVar[MaidContext] with default MaidContext() (never None)
[ ] Implement get_context() -> MaidContext — returns current context (never None, returns default empty MaidContext)
[ ] Implement bind_context(**overrides: object) -> Token[MaidContext] — copies current context with overrides, returns Token for restoring previous
[ ] Implement new_correlation_id() -> str — generates uuid4().hex[:16], binds it into current context
[ ] Implement bind_player_context(player_id: str, session_id: str) -> None — convenience for session setup
[ ] Implement clear_context() -> None — resets to empty MaidContext()
[ ] Implement _add_maid_context(logger, method_name, event_dict) -> dict structlog processor:
[ ] Single _maid_context_var.get() call
[ ] Unpacks all non-empty fields into event_dict
[ ] Write unit tests in packages/maid-engine/tests/observability/test_context.py:
[ ] Test MaidContext is frozen (immutable)
[ ] Test bind_context() sets and resets ContextVar
[ ] Test get_context() returns empty MaidContext when unset (never None)
[ ] Test structlog_context_processor injects fields
[ ] Test nested bind_context() correctly restores outer context
[ ] Test correlation_id auto-generation

1.4 ObservabilityRegistry Protocol and Implementations¶

Package: maid-engine | Priority: P0 | Dependencies: 1.2, 1.3

[ ] Define ObservabilityRegistry protocol in packages/maid-engine/src/maid_engine/observability/registry.py:
[ ] def get_meter(self) -> PrometheusMeter — returns the meter for metric creation
[ ] def get_logger(self, name: str) -> Any
[ ] def get_tracer(self, name: str) -> Any
[ ] def health_checker(self) -> HealthChecker
[ ] @property def settings(self) -> ObservabilitySettings
[ ] Implement DefaultObservabilityRegistry class:
[ ] Constructor accepts ObservabilitySettings
[ ] _meter: PrometheusMeter — single PrometheusMeter instance with caching and CardinalityGuard
[ ] get_meter() returns self._meter
[ ] get_logger() returns structlog.get_logger(name)
[ ] get_tracer() returns OpenTelemetry tracer or no-op
[ ] health_checker() returns shared HealthChecker instance
[ ] Implement NullObservabilityRegistry class:
[ ] All methods return no-op stubs (no-op meter, no-op logger, etc.)
[ ] Used when settings.observability.enabled is False
[ ] Write unit tests in packages/maid-engine/tests/observability/test_registry.py:
[ ] Test DefaultObservabilityRegistry.get_meter() returns PrometheusMeter
[ ] Test PrometheusMeter caches metric objects (same name returns same instance)
[ ] Test NullObservabilityRegistry returns no-op objects
[ ] Test CardinalityGuard enforcement (see task 1.8)

1.5 safe_observe() Context Manager¶

Package: maid-engine | Priority: P0 | Dependencies: 1.4

[ ] Implement safe_observe() context manager in packages/maid-engine/src/maid_engine/observability/safe_observe.py:
[ ] Signature: @contextmanager safe_observe(operation: str, registry: ObservabilityRegistry | None = None) -> Iterator[None]
[ ] Wraps body in try/except catching all Exception
[ ] On exception: increment maid_observability_errors_total counter with label operation
[ ] Rate-limit error logging: at most 1 log per 5 minutes (300s) per unique operation name
[ ] Use _error_timestamps: dict[str, float] module-level dict for rate limiting
[ ] Never re-raise — observability failures must not crash the game
[ ] Implement _observability_error_counter meta-counter (created lazily on first use)
[ ] Write unit tests in packages/maid-engine/tests/observability/test_safe_observe.py:
[ ] Test body executes normally when no exception
[ ] Test exception is swallowed (not re-raised)
[ ] Test error counter is incremented on exception
[ ] Test rate-limited logging (second error within 5 minutes is not logged)
[ ] Test different operations have independent rate limits

1.6 structlog Configuration and stdlib Bridge¶

Package: maid-engine | Priority: P0 | Dependencies: 1.3

[ ] Implement setup_logging(settings: ObservabilitySettings) -> None in packages/maid-engine/src/maid_engine/observability/logging.py (§6.2 of design doc):
[ ] Configure structlog with shared processor chain:
1. structlog.contextvars.merge_contextvars
2. structlog.stdlib.add_logger_name
3. structlog.stdlib.add_log_level
4. structlog.processors.TimeStamper(fmt="iso", utc=True)
5. _add_maid_context (from context.py — injects MaidContext fields via single ContextVar.get())
6. _add_log_channel — classifies events as "audit" or "operational" (§6.7)
7. structlog.processors.StackInfoRenderer()
8. structlog.processors.format_exc_info
9. structlog.processors.UnicodeDecoder()
10. If log_sampling_enabled: insert _sampling_processor at position 0
11. If anonymize_player_ids: append _anonymize_player_id processor
12. If redact_command_args: append _redact_command processor
[ ] Configure stdlib logging bridge via structlog.stdlib.ProcessorFormatter:
- [ ] foreign_pre_chain=shared_processors to capture existing logging.getLogger() calls
- [ ] Renderer: structlog.processors.JSONRenderer() when json_logs=True, else structlog.dev.ConsoleRenderer()
[ ] Set root logger level from settings.log_level
[ ] Suppress noisy loggers: asyncio, websockets, uvicorn.access → WARNING level
[ ] Configure logging.captureWarnings(True) to route Python warnings through structlog
[ ] Implement _add_log_channel(logger, method_name, event_dict) -> dict (§6.7):
[ ] _AUDIT_EVENTS: frozenset[str] — events classified as audit: "command_executed", "player_connected", "player_disconnected", "admin_action", "auth_login", "auth_failure", "ai_completion", "budget_warning", "entity_created", "entity_destroyed"
[ ] Sets event_dict["channel"] to "audit" if event name in _AUDIT_EVENTS, else "operational"
[ ] Implement _anonymize_player_id(logger, method_name, event_dict) -> dict:
[ ] If player_id key present and non-empty, replace with PlayerIDAnonymizer.anonymize() value
[ ] Write unit tests in packages/maid-engine/tests/observability/test_logging.py:
[ ] Test JSON output format includes expected fields
[ ] Test console output format is human-readable
[ ] Test stdlib log records are captured by structlog
[ ] Test _anonymize_processor replaces player IDs
[ ] Test _redact_processor redacts sensitive command arguments

1.7 Privacy: PlayerIDAnonymizer and Command Redaction¶

Package: maid-engine | Priority: P0 | Dependencies: 1.2

[ ] Implement PlayerIDAnonymizer class in packages/maid-engine/src/maid_engine/observability/privacy.py:
[ ] HMAC-based anonymization with rotating salt (§14.1 of design doc)
[ ] Constructor accepts salt_rotation_hours: int = 24
[ ] _salt: bytes — os.urandom(32) generated at init
[ ] _salt_created: float — time.monotonic() timestamp of salt creation
[ ] anonymize(player_id: str) -> str — returns f"p_{hmac.new(self._salt, player_id.encode(), hashlib.sha256).hexdigest()[:12]}"
[ ] _maybe_rotate_salt() -> None — rotates salt if elapsed time exceeds salt_rotation_hours * 3600
[ ] Same player maps to same pseudonym within rotation window (enables log correlation)
[ ] Different pseudonyms across rotation windows (GDPR-compatible)
[ ] Implement _redact_command structlog processor in privacy.py (§14.2 of design doc):
[ ] _SAFE_COMMANDS_ALLOWLIST: frozenset[str] — commands whose arguments are safe to log (loaded from ObservabilitySettings.command_args_allowlist)
[ ] Default allowlist: look, move, go, north, south, east, west, up, down, inventory, score, who, help, quit, exits, commands, areas, map, time, weather
[ ] _redact_command(logger, method_name, event_dict) -> dict — extracts verb from command field, strips args if verb not in allowlist
[ ] Whitelist approach: redact ALL arguments by default, only safe commands retain args
[ ] Content packs can register additional safe commands via register_safe_commands()
[ ] Write unit tests in packages/maid-engine/tests/observability/test_privacy.py:
[ ] Test PlayerIDAnonymizer produces deterministic output
[ ] Test PlayerIDAnonymizer produces different output for different inputs
[ ] Test PlayerIDAnonymizer cache does not exceed max size
[ ] Test CommandRedactor passes whitelisted command arguments through
[ ] Test CommandRedactor redacts non-whitelisted command arguments
[ ] Test CommandRedactor with custom whitelist

1.8 Prometheus Metrics: PrometheusMeter, CardinalityGuard, Histogram Profiles¶

Package: maid-engine | Priority: P0 | Dependencies: 1.4

[ ] Implement PrometheusMeter class in packages/maid-engine/src/maid_engine/observability/metrics.py:
[ ] Thin wrapper over prometheus_client providing prefixed metric creation
[ ] create_counter(name: str, description: str, labels: tuple[str, ...] = ()) -> Counter
[ ] create_histogram(name: str, description: str, labels: tuple[str, ...] = (), buckets: tuple[float, ...] | None = None) -> Histogram
[ ] create_gauge(name: str, description: str, labels: tuple[str, ...] = ()) -> Gauge
[ ] Internal _registry: dict[str, Counter | Histogram | Gauge] for deduplication
[ ] Auto-prefix all metric names with "maid_" (e.g., maid_tick_duration_seconds)
[ ] Implement CardinalityGuard class in metrics.py (§12.2 of design doc):
[ ] Constructor accepts caps: dict[str, int] | None = None (from ObservabilitySettings.cardinality_caps), default_cap: int = 500
[ ] _seen: dict[str, set[str]] — tracks seen label values per label name
[ ] _warned: set[str] — labels for which cap-hit warning has been emitted
[ ] check(label_name: str, value: str, allowlist: set[str] | None = None) -> str:
- [ ] If allowlist provided and value not in it: return "_other"
- [ ] If value already seen: return value
- [ ] If len(seen) >= cap: emit warning (once per label_name), return "_other"
- [ ] Otherwise: add to seen, return value
[ ] Operator-tunable per-label caps via ObservabilitySettings.cardinality_caps
[ ] Pre-defined allowlists:
- [ ] _PROVIDER_ALLOWLIST = {"anthropic", "openai", "ollama"}
- [ ] _STATUS_ALLOWLIST = {"success", "error", "rate_limited", "timeout", "not_found", "denied"}
- [ ] _AI_INTENT_ALLOWLIST = {"dialogue", "quest_gen", "narrative_desc", "combat_narrate", "npc_autonomy", "content_gen", "moderation"}
[ ] Define histogram bucket profiles per metric type in metrics.py (§4.6 of design doc):
[ ] HistogramProfile enum: COMPACT, DETAILED
[ ] TICK_BUCKETS: dict[HistogramProfile, tuple[float, ...]] — COMPACT: 6 buckets, DETAILED: 10-12
[ ] COMMAND_BUCKETS: dict[HistogramProfile, tuple[float, ...]] — command execution duration buckets
[ ] AI_LATENCY_BUCKETS: dict[HistogramProfile, tuple[float, ...]] — AI request latency buckets (0.5s–60s range)
[ ] DB_QUERY_BUCKETS: dict[HistogramProfile, tuple[float, ...]] — database query duration buckets
[ ] API_REQUEST_BUCKETS: dict[HistogramProfile, tuple[float, ...]] — HTTP API request duration buckets
[ ] Default profile: COMPACT in production, DETAILED in development (via ObservabilitySettings.histogram_profile)
[ ] Implement SystemTickAggregator class in metrics.py:
[ ] Two modes: pack_only (default) and all (opt-in via ObservabilitySettings.system_tick_detail)
[ ] pack_only: aggregates per-system tick durations into a single histogram observation per content pack
[ ] all: records per-system histogram observations (use only for debugging)
[ ] record_system(system_name: str, pack_name: str, duration: float) -> None
[ ] flush(total_tick_duration: float) -> None — observes total tick duration into histogram, resets accumulators
[ ] Write unit tests in packages/maid-engine/tests/observability/test_metrics.py:
[ ] Test PrometheusMeter creates counters with correct prefix
[ ] Test PrometheusMeter deduplicates metrics (same name returns same object)
[ ] Test CardinalityGuard allows labels within limit
[ ] Test CardinalityGuard rejects labels exceeding limit
[ ] Test CardinalityGuard substitutes "_other" on breach
[ ] Test SystemTickAggregator aggregates and flushes correctly
[ ] Test histogram profile bucket definitions

1.9 Log Sampling: AdaptiveSampler¶

Package: maid-engine | Priority: P0 | Dependencies: 1.6

[ ] Implement AdaptiveSampler class in packages/maid-engine/src/maid_engine/observability/log_sampling.py:
[ ] Constructor accepts base_rate: float (from ObservabilitySettings.operational_log_sample_rate)
[ ] should_sample(level: str, event: str, context: MaidContext | None = None) -> bool
[ ] Always sample: ERROR, CRITICAL levels (bypass sampling)
[ ] Always sample: events tagged as audit=True in event_dict
[ ] Always sample: slow-path events (tick duration > 2x target)
[ ] Apply base_rate sampling to DEBUG and INFO levels via random.random() < base_rate
[ ] _operational_counter: int — count of sampled-out operational logs for periodic summary
[ ] Implement sampling_processor(logger, method_name, event_dict) -> dict structlog processor:
[ ] Calls AdaptiveSampler.should_sample()
[ ] Raises structlog.DropEvent if not sampled
[ ] Write unit tests in packages/maid-engine/tests/observability/test_log_sampling.py:
[ ] Test ERROR level always sampled
[ ] Test CRITICAL level always sampled
[ ] Test audit events always sampled
[ ] Test INFO level sampled at configured rate
[ ] Test DEBUG level sampled at configured rate
[ ] Test sampling_processor drops events correctly

1.10 ScrapeCache with Background Rendering¶

Package: maid-engine | Priority: P0 | Dependencies: 1.8

[ ] Implement ScrapeCache class in packages/maid-engine/src/maid_engine/observability/metrics.py:
[ ] _cached_output: bytes — pre-rendered Prometheus exposition format
[ ] _last_render: float — monotonic timestamp of last render
[ ] _ttl: float — cache TTL from ObservabilitySettings.scrape_cache_ttl (default 1.0s)
[ ] _lock: asyncio.Lock — prevents concurrent rendering
[ ] async def get() -> bytes — returns cached output if within TTL, else renders fresh
[ ] async def _render() -> bytes — calls prometheus_client.generate_latest() in thread executor to avoid blocking event loop
[ ] Write unit tests in packages/maid-engine/tests/observability/test_scrape_cache.py:
[ ] Test cached output returned within TTL
[ ] Test fresh render after TTL expiry
[ ] Test concurrent calls do not trigger multiple renders

1.11 Health Checks: HealthChecker¶

Package: maid-engine | Priority: P0 | Dependencies: 1.4

[ ] Implement HealthChecker class in packages/maid-engine/src/maid_engine/observability/health.py:
[ ] HealthStatus enum: HEALTHY, DEGRADED, UNHEALTHY
[ ] HealthCheckResult dataclass: status: HealthStatus, message: str, duration_ms: float, details: dict[str, Any]
[ ] register_check(name: str, check_fn: Callable[[], Awaitable[HealthCheckResult]]) -> None
[ ] async def check_health() -> dict[str, HealthCheckResult] — runs all registered checks
[ ] async def check_liveness() -> HealthCheckResult — lightweight check (is the process alive and event loop responsive)
[ ] async def check_readiness() -> HealthCheckResult — aggregates all registered checks
[ ] Built-in checks:
- [ ] _check_event_loop() — verifies event loop is not blocked (schedules a callback, measures delay)
- [ ] _check_tick_loop(engine: GameEngine) — verifies last tick was within 5x tick interval
[ ] _db_health_cache: HealthCheckResult | None — cached DB health, updated by background loop
[ ] async def _db_health_loop(interval: float = 30.0) -> None — background task that periodically checks DB connectivity
[ ] Write unit tests in packages/maid-engine/tests/observability/test_health.py:
[ ] Test check_liveness() returns HEALTHY when event loop responsive
[ ] Test check_readiness() aggregates multiple checks
[ ] Test DEGRADED status when one check fails but others pass
[ ] Test UNHEALTHY status when critical check fails
[ ] Test DB health cache is used (not re-queried on every request)
[ ] Test custom check registration

1.12 Internal Port Server¶

Package: maid-engine | Priority: P0 | Dependencies: 1.10, 1.11

[ ] Implement InternalServer class in packages/maid-engine/src/maid_engine/observability/internal_server.py:
[ ] Lightweight ASGI application (no framework dependency — raw ASGI)
[ ] Binds to ObservabilitySettings.internal_port (default 9090)
[ ] Routes:
- [ ] GET /metrics — serves Prometheus metrics from ScrapeCache
- [ ] GET /healthz — serves overall health status (HTTP 200/503)
- [ ] GET /readyz — serves readiness status (HTTP 200/503)
- [ ] GET /livez — serves liveness status (HTTP 200/503)
[ ] Response format: JSON for health endpoints, Prometheus text exposition for /metrics
[ ] async def start() -> None — starts uvicorn or hypercorn on the internal port
[ ] async def stop() -> None — graceful shutdown
[ ] No authentication required when bound to loopback (127.0.0.1)
[ ] When internal_host is not loopback: require Bearer token from ObservabilitySettings.metrics_token; warn at startup if token is empty
[ ] Write unit tests in packages/maid-engine/tests/observability/test_internal_server.py:
[ ] Test /metrics returns valid Prometheus text format
[ ] Test /healthz returns 200 with JSON body
[ ] Test /readyz returns 503 when not ready
[ ] Test /livez returns 200 when alive
[ ] Test unknown routes return 404

1.13 Tick Loop Instrumentation¶

Package: maid-engine | Priority: P0 | Dependencies: 1.4, 1.5

[ ] Modify GameEngine._tick_loop() in packages/maid-engine/src/maid_engine/core/engine.py:
[ ] Wrap tick body in with safe_observe("tick_loop"):
[ ] Record tick duration: self._obs_registry.histogram("tick_duration_seconds", "Duration of a single game tick").observe(duration)
[ ] Increment tick counter: self._obs_registry.counter("ticks_total", "Total number of ticks processed").inc()
[ ] Record per-system durations via SystemTickAggregator if tracing_mode >= "default"
[ ] Set maid_tick_loop_healthy gauge: 1.0 if last_tick_duration < tick_budget, 0.0 otherwise (referenced by SLO burn-rate alerts)
[ ] Bind MaidContext(tick_number=self.tick_count) for the tick scope
[ ] Add last_tick_monotonic: float property to GameEngine:
[ ] Set self._last_tick_monotonic = time.monotonic() at start of each tick
[ ] Used by HealthChecker._check_tick_loop() to detect stalled tick loops
[ ] Write unit tests in packages/maid-engine/tests/observability/test_tick_instrumentation.py:
[ ] Test tick duration histogram is observed
[ ] Test tick counter is incremented
[ ] Test last_tick_monotonic is updated each tick
[ ] Test safe_observe swallows observability errors without affecting tick

1.14 Command Instrumentation¶

Package: maid-engine | Priority: P0 | Dependencies: 1.3, 1.4, 1.5

[ ] Modify LayeredCommandRegistry.execute() in packages/maid-engine/src/maid_engine/commands/registry.py:
[ ] Bind MaidContext(command=command_name, player_id=context.player_id, correlation_id=uuid4()) at command entry
[ ] Wrap execution in with safe_observe("command_execute"):
[ ] Record command duration: registry.histogram("command_duration_seconds", "Command execution duration", labels=("command",)).labels(command=command_name).observe(duration)
[ ] Increment command counter: registry.counter("commands_total", "Total commands processed", labels=("command", "status")).labels(command=command_name, status="ok"|"error").inc()
[ ] The ObservabilityRegistry reference comes from CommandContext — add obs_registry: ObservabilityRegistry | None = None field to CommandContext
[ ] Write unit tests in packages/maid-engine/tests/observability/test_command_instrumentation.py:
[ ] Test command duration histogram is observed with correct label
[ ] Test command counter incremented on success
[ ] Test command counter incremented with status="error" on failure
[ ] Test MaidContext is set with correct command name and player_id
[ ] Test instrumentation is no-op when obs_registry is None

1.15 EventBus Instrumentation¶

Package: maid-engine | Priority: P0 | Dependencies: 1.4, 1.5

[ ] Modify EventBus.emit() in packages/maid-engine/src/maid_engine/core/events.py:
[ ] Wrap in with safe_observe("event_emit"):
[ ] Increment event counter: registry.counter("events_total", "Total events emitted", labels=("event_domain",)).labels(event_domain=get_event_domain(event)).inc()
[ ] For emit_sync() deferred events: capture current MaidContext at queue time and restore it when the handler runs, to preserve correlation IDs across async boundaries
[ ] get_event_domain(event_type_name: str) -> str — maps event class names to ~15 bounded domain labels via prefix matching (§12.1 of design doc):
- [ ] _EVENT_DOMAIN_MAP dict mapping prefixes to domains: Tick/Startup/Shutdown → "engine", Connect/Disconnect → "network", Room/Move → "movement", Combat/Damage/Death → "combat", Item/Pickup/Drop → "inventory", Spell/Cast → "magic", Quest → "quest", Guild/Faction → "social", Auction/Trade → "economy", Craft → "crafting", NPC/Dialogue → "npc", PvP → "pvp", Weather/Time → "world"
- [ ] Unknown event types → "other" fallback
[ ] Add obs_registry: ObservabilityRegistry | None attribute to EventBus.__init__():
[ ] Default None — no instrumentation unless explicitly set
[ ] Set by GameEngine during initialization
[ ] Write unit tests in packages/maid-engine/tests/observability/test_event_instrumentation.py:
[ ] Test event counter incremented on emit
[ ] Test domain bucketing maps event types correctly
[ ] Test unknown event types map to "other" bucket
[ ] Test instrumentation is no-op when obs_registry is None

1.16 O(1) Entity Counting via EntityManager¶

Package: maid-engine | Priority: P0 | Dependencies: none

[ ] Add count_with_tag(tag: str) -> int method to EntityManager in packages/maid-engine/src/maid_engine/core/ecs/entity.py:
[ ] Leverages existing _by_tag index for O(1) lookup via len(self._by_tag.get(tag, set()))
[ ] Expose total entity count via existing _entities dict: @property entity_count(self) -> int
[ ] Add convenience methods to World in packages/maid-engine/src/maid_engine/core/world.py:
[ ] @property entity_count(self) -> int — delegates to self._entity_manager.entity_count
[ ] def entity_type_count(self, entity_type: str) -> int — delegates to self._entity_manager.count_with_tag(entity_type)
[ ] These counters feed Prometheus gauges without requiring O(N) iteration
[ ] Write unit tests in packages/maid-engine/tests/observability/test_entity_counting.py:
[ ] Test count increments on create
[ ] Test count decrements on destroy
[ ] Test type-specific counts
[ ] Test count is 0 after creating and destroying same entity

1.17 GameEngine Integration¶

Package: maid-engine | Priority: P0 | Dependencies: 1.4, 1.11, 1.12, 1.13

[ ] Modify GameEngine.__init__() in packages/maid-engine/src/maid_engine/core/engine.py:
[ ] Create ObservabilityRegistry based on settings.observability.enabled:
- If enabled: self._obs_registry = DefaultObservabilityRegistry(settings.observability)
- If disabled: self._obs_registry = NullObservabilityRegistry()
[ ] Store as self._obs_registry: ObservabilityRegistry
[ ] Expose as @property obs_registry(self) -> ObservabilityRegistry
[ ] Pass obs_registry to EventBus via self.world.events.obs_registry = self._obs_registry
[ ] Modify GameEngine.start():
[ ] Call setup_logging(settings.observability) before other initialization
[ ] Register built-in health checks on self._obs_registry.health_checker()
[ ] Start InternalServer on settings.observability.internal_port
[ ] Store self._internal_server: InternalServer
[ ] Modify GameEngine.stop():
[ ] Stop InternalServer gracefully
[ ] Write unit tests in packages/maid-engine/tests/observability/test_engine_integration.py:
[ ] Test GameEngine creates DefaultObservabilityRegistry when enabled
[ ] Test GameEngine creates NullObservabilityRegistry when disabled
[ ] Test obs_registry property returns the registry
[ ] Test health checks are registered on start

1.18 ASGI Middleware¶

Package: maid-engine | Priority: P1 | Dependencies: 1.3, 1.4

[ ] Implement ObservabilityMiddleware ASGI middleware in packages/maid-engine/src/maid_engine/observability/middleware.py:
[ ] Wraps the existing ASGI application (WebServer.app)
[ ] On each request:
- [ ] Extract path_template from request path (collapse path params to {id} to limit cardinality)
- [ ] Bind MaidContext(correlation_id=uuid4()) for the request scope
- [ ] Record request duration: histogram("http_request_duration_seconds", labels=("method", "path_template", "status_code")).observe(duration)
- [ ] Increment request counter: counter("http_requests_total", labels=("method", "path_template", "status_code")).inc()
[ ] _extract_path_template(path: str) -> str — e.g., /admin/entities/abc-123 → /admin/entities/{id}
[ ] Register middleware on WebServer in packages/maid-engine/src/maid_engine/net/web/server.py:
[ ] Wrap self._app with ObservabilityMiddleware if observability is enabled
[ ] Write unit tests in packages/maid-engine/tests/observability/test_middleware.py:
[ ] Test path template extraction collapses UUIDs
[ ] Test request duration histogram is observed
[ ] Test request counter is incremented with correct labels
[ ] Test MaidContext is bound for request scope

1.19 Benchmark Validation Gate¶

Package: maid-engine | Priority: P0 | Dependencies: 1.13, 1.14, 1.15, 1.16

This is a validation gate — Phase 2 must not begin until these benchmarks pass.

[ ] Create benchmark script at packages/maid-engine/tests/observability/bench_overhead.py:
[ ] Measure tick loop overhead with observability enabled vs disabled
[ ] Target: < 2% overhead on tick loop at 4 ticks/second
[ ] Measure command execution overhead with instrumentation
[ ] Target: < 1ms additional latency per command
[ ] Measure EventBus emit overhead with instrumentation
[ ] Target: < 0.1ms additional latency per event
[ ] Measure memory overhead of ObservabilityRegistry
[ ] Target: < 50MB additional memory at 10,000 entities
[ ] Document benchmark results in docs/impl/v3.1/06-benchmarks.md

Phase 2: AI and Game Metrics (Weeks 5–8) — P0/P1¶

2.1 CompletionChunk and TokenUsage Dataclasses¶

Package: maid-engine | Priority: P0 | Dependencies: none

[ ] Add CompletionChunk frozen dataclass to packages/maid-engine/src/maid_engine/ai/providers/base.py:
[ ] content: str — text chunk from streaming response
[ ] finish_reason: str | None = None — set on final chunk
[ ] model: str | None = None — set on final chunk
[ ] usage: TokenUsage | None = None — set on final chunk (if provider supports it)
[ ] Add TokenUsage frozen dataclass to base.py:
[ ] prompt_tokens: int
[ ] completion_tokens: int
[ ] total_tokens: int
[ ] Write unit tests in packages/maid-engine/tests/ai/test_completion_chunk.py:
[ ] Test CompletionChunk is frozen
[ ] Test TokenUsage total equals prompt + completion

2.2 LLMProvider Base Class Refactor with Instrumentation¶

Package: maid-engine | Priority: P0 | Dependencies: 1.4, 2.1

[ ] Refactor LLMProvider in packages/maid-engine/src/maid_engine/ai/providers/base.py:
[ ] Rename existing abstract complete() to _do_complete(self, messages: list[Message], options: CompletionOptions) -> CompletionResult (abstract)
[ ] Add new concrete complete() wrapper method:
- [ ] Calls _do_complete() internally
- [ ] Wraps in with safe_observe("llm_complete"):
- [ ] Records maid_ai_request_duration_seconds histogram with labels (provider, model)
- [ ] Records maid_ai_tokens_total counter with labels (provider, model, token_type) where token_type is "prompt" or "completion"
- [ ] Records maid_ai_requests_total counter with labels (provider, model, status) where status is "ok" or "error"
- [ ] Emits AICompletionEvent via EventBus (if available)
[ ] Add obs_registry: ObservabilityRegistry | None = None attribute to LLMProvider.__init__()
[ ] Add event_bus: EventBus | None = None attribute to LLMProvider.__init__()
[ ] Update all three providers — each currently has a concrete complete() override:
[ ] AnthropicProvider: rename complete() → _do_complete(), ensure TokenUsage populated from API response
[ ] OpenAIProvider: rename complete() → _do_complete(), ensure TokenUsage populated from API response
[ ] OllamaProvider: rename complete() → _do_complete(), ensure TokenUsage populated from API response
[ ] Update provider construction in ai/registry.py (create_registry_from_settings):
[ ] Pass obs_registry and event_bus to provider constructors
[ ] Update all complete_streaming() consumers across packages (breaking change):
[ ] Search complete_streaming in maid-engine, maid-classic-rpg, and maid-stdlib
[ ] Update maid_classic_rpg/systems/npc/dialogue.py and test doubles in tests/test_npc_dialogue_system.py
[ ] Update consumers to access chunk.content for text instead of bare str
[ ] Write unit tests in packages/maid-engine/tests/observability/test_llm_instrumentation.py:
[ ] Test complete() wrapper records maid_ai_request_duration_seconds histogram
[ ] Test complete() wrapper records maid_ai_tokens_total counter
[ ] Test complete() wrapper records maid_ai_requests_total counter with status
[ ] Test complete() wrapper emits AICompletionEvent
[ ] Test instrumentation is no-op when obs_registry is None
[ ] Test cross-package: maid-classic-rpg NPC dialogue system works with refactored providers

2.3 Streaming Response Fix with CompletionChunk¶

Package: maid-engine | Priority: P0 | Dependencies: 2.1

[ ] Refactor LLMProvider.complete_streaming() in packages/maid-engine/src/maid_engine/ai/providers/base.py:
[ ] Change return type from AsyncIterator[str] to AsyncIterator[CompletionChunk]
[ ] Rename existing complete_streaming() to _do_complete_streaming() (abstract)
[ ] Add wrapper complete_streaming() that:
- [ ] Wraps iteration in try/finally for cancellation safety
- [ ] Records total streaming duration on completion
- [ ] Records token usage from final chunk (if available)
- [ ] Increments llm_streaming_requests_total counter
[ ] Update AnthropicProvider.complete_streaming() → _do_complete_streaming():
[ ] Yield CompletionChunk objects with content and final-chunk metadata
[ ] Update OpenAIProvider.complete_streaming() → _do_complete_streaming():
[ ] Yield CompletionChunk objects with content and final-chunk metadata
[ ] Update OllamaProvider.complete_streaming() → _do_complete_streaming():
[ ] Yield CompletionChunk objects with content and final-chunk metadata
[ ] Update all call sites that consume complete_streaming() to handle CompletionChunk instead of str:
[ ] Search for complete_streaming usage across all packages
[ ] Update to access chunk.content for text
[ ] Write unit tests in packages/maid-engine/tests/ai/test_streaming_fix.py:
[ ] Test streaming returns CompletionChunk objects
[ ] Test final chunk includes finish_reason and usage
[ ] Test cancellation safety (async generator cleanup)
[ ] Test duration is recorded on stream completion

2.4 AI Cost Calculation and Pricing¶

Package: maid-engine | Priority: P0 | Dependencies: 2.1

[ ] Implement AI cost calculation in packages/maid-engine/src/maid_engine/observability/ai_metrics.py:
[ ] PricingEntry dataclass: model: str, prompt_cost_per_1k: float, completion_cost_per_1k: float, effective_date: str
[ ] PricingConfig dataclass: entries: dict[str, PricingEntry], default_prompt_cost: float = 0.0, default_completion_cost: float = 0.0
[ ] load_pricing(config_path: Path | None = None, env_override: str | None = None) -> PricingConfig:
- [ ] Priority: env var → config file → hardcoded defaults
- [ ] Config file path: data/ai_pricing.yml (YAML format)
- [ ] Env var: MAID_AI__PRICING_JSON (JSON string)
- [ ] DocumentStore persistence and runtime admin API deferred to Phase 3
[ ] calculate_cost(usage: TokenUsage, model: str, pricing: PricingConfig) -> float:
- [ ] Returns cost in USD
- [ ] Uses model-specific pricing if available, else default
[ ] Implement AICompletionEvent event class in ai_metrics.py:
[ ] Extends Event base class
[ ] Fields: provider: str, model: str, usage: TokenUsage, duration: float, cost: float, player_id: str | None
[ ] Implement AICostTracker class in ai_metrics.py:
[ ] Event-driven: subscribes to AICompletionEvent via EventBus
[ ] Maintains running totals: _total_cost: float, _total_tokens: int, _costs_by_model: dict[str, float], _costs_by_player: dict[str, float]
[ ] get_total_cost() -> float
[ ] get_cost_by_model() -> dict[str, float]
[ ] get_cost_by_player() -> dict[str, float]
[ ] reset() -> None — resets all counters (called by gauge export loop)
[ ] Exposes Prometheus gauges: maid_ai_cost_total_usd, maid_ai_cost_by_model_usd, maid_ai_tokens_total
[ ] Write unit tests in packages/maid-engine/tests/observability/test_ai_metrics.py:
[ ] Test load_pricing() from file
[ ] Test load_pricing() env var overrides file
[ ] Test calculate_cost() with known model pricing
[ ] Test calculate_cost() with default pricing
[ ] Test AICostTracker accumulates costs from events
[ ] Test AICostTracker tracks per-model and per-player costs
[ ] Test AICostTracker.reset() clears accumulators

2.5 Runtime Pricing Admin API¶

Package: maid-engine | Priority: P2 | Dependencies: 2.4 Note: Deferred to Phase 3. Phase 2 load_pricing() uses env var → config file → defaults only. DocumentStore persistence added here.

[ ] Add PUT /admin/ai/pricing endpoint in packages/maid-engine/src/maid_engine/api/admin/:
[ ] Accepts JSON body with pricing entries
[ ] Validates entries via PricingConfig Pydantic model
[ ] Persists to DocumentStore under collection "ai_config"
[ ] Reloads AICostTracker pricing on update
[ ] Requires AccessLevel.ADMIN
[ ] Add GET /admin/ai/pricing endpoint:
[ ] Returns current pricing configuration
[ ] Write unit tests in packages/maid-engine/tests/api/test_ai_pricing_api.py:
[ ] Test PUT updates pricing
[ ] Test GET returns current pricing
[ ] Test validation rejects invalid pricing entries
[ ] Test admin access level required

2.6 Rate Limiter Instrumentation¶

Package: maid-engine | Priority: P1 | Dependencies: 1.4

[ ] Modify RateLimiter.check_and_reserve() in packages/maid-engine/src/maid_engine/ai/rate_limiter.py:
[ ] Add obs_registry: ObservabilityRegistry | None = None to RateLimiter.__init__()
[ ] On denial: increment maid_ai_rate_limit_denials_total counter with labels (reason,) where reason is "global_rpm", "player_rpm", "daily_budget", "player_daily_budget"
[ ] On allow: increment maid_ai_rate_limit_allowed_total counter
[ ] Wrap in with safe_observe("rate_limit_check"):
[ ] Write unit tests in packages/maid-engine/tests/observability/test_rate_limiter_metrics.py:
[ ] Test denial counter incremented with correct reason label
[ ] Test allowed counter incremented on successful reservation
[ ] Test no-op when obs_registry is None

2.7 Circuit Breaker Metrics¶

Package: maid-engine | Priority: P1 | Dependencies: 1.4

[ ] Modify CircuitBreaker in packages/maid-engine/src/maid_engine/ai/circuit_breaker.py:
[ ] Add obs_registry: ObservabilityRegistry | None = None to CircuitBreaker.__init__()
[ ] On state transition: set maid_ai_circuit_breaker_state gauge with labels (provider,) to 0 (CLOSED), 1 (OPEN), 2 (HALF_OPEN)
[ ] On trip: increment maid_ai_circuit_breaker_trips_total counter with labels (provider,)
[ ] Wrap state transitions in with safe_observe("circuit_breaker"):
[ ] Write unit tests in packages/maid-engine/tests/observability/test_circuit_breaker_metrics.py:
[ ] Test state gauge updated on transition
[ ] Test trips counter incremented on trip
[ ] Test no-op when obs_registry is None

2.8 MetricsCollector Unification¶

Package: maid-engine | Priority: P1 | Dependencies: 1.8, 2.4

[ ] Refactor MetricsCollector in packages/maid-engine/src/maid_engine/api/admin/dashboard.py:
[ ] Where possible, read from existing Prometheus Gauge/Counter objects rather than re-computing
[ ] collect_server_metrics() reads maid_tick_duration_seconds histogram for tick stats
[ ] collect_player_metrics() reads entity count gauges for player counts
[ ] collect_world_metrics() reads entity type count gauges
[ ] Retain computed metrics for data not exposed via Prometheus (e.g., detailed per-room counts)
[ ] Add collect_ai_metrics() -> AIMetrics that reads from AICostTracker gauges
[ ] Add AIMetrics dataclass: total_cost: float, total_tokens: int, costs_by_model: dict[str, float], active_providers: list[str]
[ ] Write unit tests in packages/maid-engine/tests/observability/test_metrics_collector_unified.py:
[ ] Test collect_server_metrics() reads from Prometheus
[ ] Test collect_ai_metrics() returns cost data
[ ] Test unified collector returns consistent data with Prometheus exposition

2.9 Database Metrics¶

Package: maid-engine | Priority: P1 | Dependencies: 1.4

[ ] Add database instrumentation hooks to packages/maid-engine/src/maid_engine/storage/document_store.py:
[ ] maid_db_query_duration_seconds histogram with labels (operation, collection) where operation is "get", "get_many", "create", "update", "delete", "query", "count"
[ ] maid_db_operations_total counter with labels (operation, collection, status)
[ ] maid_db_connections_active gauge for active connection count (PostgreSQL only)
[ ] Instrument PostgresDocumentCollection methods: get(), get_many(), create(), update(), delete(), query(), count()
[ ] Wrap instrumentation in with safe_observe("db_operation"):
[ ] Add obs_registry: ObservabilityRegistry | None = None to DocumentStore.__init__() and propagate to collections
[ ] Write unit tests in packages/maid-engine/tests/observability/test_db_metrics.py:
[ ] Test query duration histogram observed on get
[ ] Test operations counter incremented with correct labels
[ ] Test no-op when obs_registry is None

2.10 Network Metrics¶

Package: maid-engine | Priority: P1 | Dependencies: 1.4

[ ] Add network instrumentation to packages/maid-engine/src/maid_engine/net/web/server.py:
[ ] maid_net_connections_active gauge with labels (protocol,) where protocol is "telnet", "websocket"
[ ] maid_net_bytes_sent_total counter with labels (protocol,)
[ ] maid_net_bytes_received_total counter with labels (protocol,)
[ ] maid_net_messages_sent_total counter with labels (protocol,)
[ ] maid_net_messages_received_total counter with labels (protocol,)
[ ] Increment on send/receive in WebSocket handler
[ ] Add network instrumentation to packages/maid-engine/src/maid_engine/net/telnet/:
[ ] Same counter/gauge pattern for telnet connections
[ ] Write unit tests in packages/maid-engine/tests/observability/test_network_metrics.py:
[ ] Test connection gauge incremented on connect, decremented on disconnect
[ ] Test bytes counters incremented on send/receive
[ ] Test message counters incremented correctly

2.11 Sentry Integration (Optional)¶

Package: maid-engine | Priority: P2 | Dependencies: 1.4

[ ] Implement optional Sentry integration in packages/maid-engine/src/maid_engine/observability/sentry.py:
[ ] configure_sentry(settings: ObservabilitySettings) -> None:
- [ ] Guard: return immediately if sentry-sdk is not installed or sentry_dsn is empty
- [ ] Call sentry_sdk.init(dsn=settings.sentry_dsn, traces_sample_rate=settings.sentry_traces_sample_rate)
- [ ] Configure before_send hook to add game context (tick number, player count, active commands)
- [ ] Configure before_send_transaction hook for trace sampling
[ ] capture_game_exception(exc: Exception, context: MaidContext | None = None) -> None:
- [ ] Adds MaidContext fields as Sentry tags
- [ ] Adds game state snapshot as Sentry extra data
[ ] Write unit tests in packages/maid-engine/tests/observability/test_sentry.py:
[ ] Test configure_sentry() is no-op when sentry-sdk not installed
[ ] Test configure_sentry() is no-op when sentry_dsn is empty
[ ] Test capture_game_exception() includes MaidContext tags (mock sentry_sdk)

Phase 3: Tracing, Dashboards, and Polish (Weeks 9–11) — P1/P2¶

3.1 OpenTelemetry Tracing Setup¶

Package: maid-engine | Priority: P1 | Dependencies: 1.4

[ ] Implement tracing setup in packages/maid-engine/src/maid_engine/observability/tracing.py:
[ ] configure_tracing(settings: ObservabilitySettings) -> Tracer:
- [ ] If opentelemetry not installed: return NoOpTracer() (from opentelemetry.trace or custom stub)
- [ ] If installed but tracing_enabled is False: return NoOpTracer()
- [ ] Configure TracerProvider with BatchSpanProcessor
- [ ] If otlp extra installed: configure OTLPSpanExporter with endpoint from env var OTEL_EXPORTER_OTLP_ENDPOINT
- [ ] Set service name: maid-engine
- [ ] Set resource attributes: service.version, deployment.environment
[ ] NoOpTracer class:
- [ ] start_span(name, **kwargs) -> NoOpSpan returns context manager that does nothing
- [ ] Used when OpenTelemetry is not available
[ ] TracingMode enum: MINIMAL, DEFAULT, VERBOSE
- [ ] MINIMAL: only command root spans
- [ ] DEFAULT: commands + AI calls + storage operations
- [ ] VERBOSE: all of above + event handlers + individual system ticks
[ ] Write unit tests in packages/maid-engine/tests/observability/test_tracing.py:
[ ] Test NoOpTracer returns no-op spans
[ ] Test configure_tracing() returns NoOpTracer when OTel not installed
[ ] Test configure_tracing() returns NoOpTracer when tracing disabled
[ ] Test TracingMode enum values

3.2 Command and AI Tracing¶

Package: maid-engine | Priority: P1 | Dependencies: 3.1, 1.14, 2.2

[ ] Add tracing to LayeredCommandRegistry.execute() in packages/maid-engine/src/maid_engine/commands/registry.py:
[ ] Create root span command:{command_name} with attributes command, player_id, correlation_id
[ ] Only when tracing_mode >= TracingMode.MINIMAL
[ ] Add tracing to LLMProvider.complete() wrapper in packages/maid-engine/src/maid_engine/ai/providers/base.py:
[ ] Create child span llm.complete with attributes provider, model, prompt_tokens, completion_tokens
[ ] Only when tracing_mode >= TracingMode.DEFAULT
[ ] Add tracing to EventBus.emit() in packages/maid-engine/src/maid_engine/core/events.py:
[ ] Create child span event:{event_type} with attributes event_type, handler_count
[ ] Only when tracing_mode >= TracingMode.VERBOSE
[ ] Add tracing to DocumentStore operations:
[ ] Create child span db:{operation} with attributes collection, operation
[ ] Only when tracing_mode >= TracingMode.DEFAULT
[ ] Write unit tests in packages/maid-engine/tests/observability/test_command_tracing.py:
[ ] Test root span created for command execution
[ ] Test child span created for AI completion
[ ] Test no spans when tracing_mode is too low
[ ] Test span attributes are correctly set

3.3 Grafana Dashboard Definitions¶

Package: maid-engine | Priority: P1 | Dependencies: 1.8, 2.4

[ ] Create docs/dashboards/maid-server-health.json:
[ ] Grafana dashboard JSON definition for Server Health
[ ] Panels:
- [ ] Tick loop duration histogram (p50, p95, p99)
- [ ] Tick rate gauge (ticks/second)
- [ ] Commands per second rate
- [ ] Command latency by command type
- [ ] Active connections by protocol (telnet/websocket)
- [ ] Entity counts by type
- [ ] Event emission rate by domain bucket
- [ ] Memory usage gauge
- [ ] Database query latency (p50, p95, p99)
- [ ] Health check status
[ ] Variables: ,
[ ] Refresh interval: 10s
[ ] Create docs/dashboards/maid-ai-cost-overview.json:
[ ] Grafana dashboard JSON definition for AI Cost Overview
[ ] Panels:
- [ ] Total AI cost (USD) over time
- [ ] Cost by model (stacked area)
- [ ] Token usage by model (stacked area)
- [ ] Request rate by provider
- [ ] Request latency by provider (p50, p95, p99)
- [ ] Rate limiter denial rate
- [ ] Circuit breaker state timeline
- [ ] Cost per player (top 10)
- [ ] Daily token budget remaining
[ ] Variables: ,, ``
[ ] Refresh interval: 30s

3.4 Content Pack Metric API¶

Package: maid-engine | Priority: P1 | Dependencies: 1.4, 1.8

[ ] Add content pack metric registration to ObservabilityRegistry in packages/maid-engine/src/maid_engine/observability/registry.py:
[ ] register_pack_counter(pack_name: str, name: str, description: str, labels: tuple[str, ...] = ()) -> Counter:
- [ ] Validates prefix: metric name must start with pack_name + "_" or be pack_name itself
- [ ] Validates labels against denylist (no player_id, session_id, ip_address)
- [ ] Full metric name: maid_pack_{pack_name}_{name}
- [ ] Subject to CardinalityGuard
- [ ] Handle name collisions: if metric already registered (e.g., two packs with same prefix), return existing metric and log a warning
[ ] register_pack_histogram(pack_name: str, name: str, description: str, labels: tuple[str, ...] = (), buckets: tuple[float, ...] | None = None) -> Histogram:
- [ ] Same validation as register_pack_counter()
[ ] register_pack_gauge(pack_name: str, name: str, description: str, labels: tuple[str, ...] = ()) -> Gauge:
- [ ] Same validation as register_pack_counter()
[ ] LABEL_DENYLIST: frozenset[str] = frozenset({"player_id", "session_id", "ip_address", "email", "password"}) for labels content packs may not use
[ ] Write unit tests in packages/maid-engine/tests/observability/test_pack_metrics.py:
[ ] Test register_pack_counter() creates metric with correct prefix
[ ] Test prefix validation rejects metrics not starting with pack name
[ ] Test label denylist rejects forbidden labels
[ ] Test CardinalityGuard is enforced for pack metrics

3.5 structlog Migration (Tier 1-2 Core Modules)¶

Package: maid-engine | Priority: P1 | Dependencies: 1.6

[ ] Migrate the following modules from logging.getLogger() to structlog.get_logger():
[ ] packages/maid-engine/src/maid_engine/core/engine.py
[ ] packages/maid-engine/src/maid_engine/core/world.py
[ ] packages/maid-engine/src/maid_engine/core/events.py
[ ] packages/maid-engine/src/maid_engine/commands/registry.py
[ ] packages/maid-engine/src/maid_engine/ai/providers/base.py
[ ] packages/maid-engine/src/maid_engine/ai/rate_limiter.py
[ ] packages/maid-engine/src/maid_engine/ai/circuit_breaker.py
[ ] packages/maid-engine/src/maid_engine/net/web/server.py
[ ] packages/maid-engine/src/maid_engine/storage/document_store.py
[ ] packages/maid-engine/src/maid_engine/plugins/loader.py
[ ] Migration pattern for each module:
[ ] Replace import logging / logger = logging.getLogger(__name__) with import structlog / logger = structlog.get_logger(__name__)
[ ] Replace logger.info("message %s", arg) with logger.info("message", arg=arg)
[ ] Add structured context where beneficial (e.g., logger.info("entity_created", entity_id=str(entity.id)))
[ ] Verify all existing tests pass after migration

3.6 Profiling Bridge — DROPPED¶

Dropped: The always-on Prometheus instrumentation (tick histograms, command counters, etc.) already covers the same data at higher quality than the session-based profiling bridge. The ProfilingBridge would only produce data when an admin runs @profile start, which is rare. Operators can correlate profiling sessions with Prometheus data via timestamps if needed.

3.7 Gauge Export Loop¶

Package: maid-engine | Priority: P1 | Dependencies: 1.16, 2.4

[ ] Implement gauge export background task in packages/maid-engine/src/maid_engine/observability/hooks.py:
[ ] async def gauge_export_loop(engine: GameEngine, interval: float = 15.0) -> None:
- [ ] Periodically updates Prometheus gauges:
- [ ] maid_entities_total gauge with labels (type,) from World.entity_type_count()
- [ ] maid_players_online gauge from active session count
- [ ] maid_rooms_total gauge from World.entity_type_count("room")
- [ ] maid_uptime_seconds gauge from GameEngine.uptime
- [ ] Calls AICostTracker cleanup for stale per-player entries
[ ] Started by GameEngine.start() as an asyncio.Task
[ ] Cancelled by GameEngine.stop()
[ ] Write unit tests in packages/maid-engine/tests/observability/test_gauge_export.py:
[ ] Test entity gauges updated from World counters
[ ] Test player online gauge reflects active sessions
[ ] Test loop runs at configured interval
[ ] Test loop handles exceptions gracefully (continues running)

3.8 Client-Side Error Ingestion — DROPPED¶

Dropped: No production web client exists to consume this endpoint. Adding a public-facing endpoint with PII sanitization and rate limiting for a theoretical future consumer is premature. Reintroduce when a web client ships.

3.9 SLO/SLI Definitions and Alert Rules¶

Package: maid-engine | Priority: P1 | Dependencies: 1.8

[ ] Create deploy/prometheus/maid_alert_rules.yml.j2 (Jinja2 template):
[ ] SLO: Tick loop latency p99 < 250ms
- [ ] MaidTickLoopSlow warning when p99 > 200ms for 5 minutes
- [ ] MaidTickLoopStalled critical when no tick for > 10 seconds
[ ] SLO: Command latency p99 < 500ms
- [ ] MaidCommandSlow warning when p99 > 400ms for 5 minutes
[ ] SLO: AI request error rate < 5%
- [ ] MaidAIErrorRateHigh warning when error rate > 3% for 10 minutes
- [ ] MaidAIErrorRateCritical critical when error rate > 10% for 5 minutes
[ ] SLO: AI daily budget
- [ ] MaidAIBudgetWarning warning when daily spend > 80% of budget
- [ ] MaidAIBudgetExhausted critical when daily spend > 95% of budget
[ ] Infrastructure:
- [ ] MaidTargetDown critical when target is down for > 2 minutes
- [ ] MaidHighMemory warning when RSS > 80% of limit
[ ] Multi-window burn-rate alerting for SLOs:
- [ ] 1h window: fast burn (14.4x error budget)
- [ ] 6h window: slow burn (6x error budget)
[ ] Create deploy/prometheus/maid_alert_config.yml:
[ ] Template variables for customization: tick_rate, command_latency_target, ai_budget_daily_usd
[ ] Create deploy/prometheus/alertmanager_routing.yml:
[ ] Route critical alerts to PagerDuty/Slack
[ ] Route warning alerts to Slack only
[ ] Group by alertname with 5-minute group wait

3.10 Runbooks¶

Package: documentation | Priority: P1 | Dependencies: 3.9

[ ] Create docs/runbooks/MaidTickLoopStalled.md:
[ ] Symptom description
[ ] Diagnostic steps (check tick duration histogram, check system timing, check event loop blocking)
[ ] Common causes (blocking I/O in system, large entity count, database timeout)
[ ] Remediation steps
[ ] Create docs/runbooks/MaidAIBudgetExhausted.md:
[ ] Symptom description
[ ] Diagnostic steps (check AI cost gauges, check per-model costs, check per-player costs)
[ ] Common causes (runaway NPC dialogue, misconfigured rate limits, pricing change)
[ ] Remediation steps (adjust rate limits, disable specific providers, increase budget)
[ ] Create docs/runbooks/MaidTargetDown.md:
[ ] Symptom description
[ ] Diagnostic steps (check health endpoints, check process status, check logs)
[ ] Common causes (OOM crash, unhandled exception, port conflict)
[ ] Remediation steps

3.11 Operator Guide Documentation¶

Package: documentation | Priority: P2 | Dependencies: 3.3, 3.9, 3.10

[ ] Create docs/guides/operator-observability.md:
[ ] Overview of observability architecture
[ ] Configuration reference for all MAID_OBSERVABILITY__* environment variables
[ ] Profile descriptions (dev/staging/prod)
[ ] Prometheus scrape configuration example
[ ] Grafana dashboard import instructions
[ ] Alert rule deployment instructions
[ ] structlog configuration and log format reference
[ ] Health endpoint documentation
[ ] Tracing setup with OpenTelemetry Collector
[ ] Sentry integration guide
[ ] Content pack metric API usage for pack authors
[ ] Log rotation configuration: logrotate examples for audit logs (90-day retention) and operational logs (14-day retention)

3.12 Alertmanager Routing Template¶

Package: maid-engine | Priority: P2 | Dependencies: 3.9

[ ] Create deploy/alertmanager/maid_routes.yml.j2:
[ ] Jinja2 template for Alertmanager routing configuration
[ ] Variables: slack_webhook_url, pagerduty_service_key, email_to
[ ] Routes:
- [ ] severity: critical to PagerDuty + Slack
- [ ] severity: warning to Slack only
- [ ] alertname: MaidAIBudgetWarning to dedicated AI cost channel
[ ] Inhibition rules: suppress warning when critical is firing for same alertname

Files Created (Summary)¶

packages/maid-engine/src/maid_engine/observability/
    __init__.py
    registry.py           # ObservabilityRegistry protocol + Default + Null
    logging.py            # structlog config, processors, stdlib bridge
    metrics.py            # PrometheusMeter, ScrapeCache, CardinalityGuard, SystemTickAggregator
    tracing.py            # OTel tracer setup (no-op if not installed)
    context.py            # MaidContext frozen dataclass, ContextVar
    health.py             # HealthChecker, HealthStatus, HealthCheckResult
    internal_server.py    # Dedicated internal port ASGI server
    middleware.py         # ASGI ObservabilityMiddleware
    ai_metrics.py         # PricingConfig, AICostTracker, AICompletionEvent, calculate_cost()
    hooks.py              # gauge_export_loop() background task
    safe_observe.py       # safe_observe() context manager
    privacy.py            # PlayerIDAnonymizer, CommandRedactor
    sentry.py             # Optional Sentry integration
    log_sampling.py       # AdaptiveSampler, sampling_processor
    bridges/
        __init__.py

packages/maid-engine/tests/observability/
    __init__.py
    test_settings.py
    test_context.py
    test_registry.py
    test_safe_observe.py
    test_logging.py
    test_privacy.py
    test_metrics.py
    test_log_sampling.py
    test_scrape_cache.py
    test_health.py
    test_internal_server.py
    test_tick_instrumentation.py
    test_command_instrumentation.py
    test_event_instrumentation.py
    test_entity_counting.py
    test_engine_integration.py
    test_middleware.py
    test_llm_instrumentation.py
    test_ai_metrics.py
    test_rate_limiter_metrics.py
    test_circuit_breaker_metrics.py
    test_metrics_collector_unified.py
    test_db_metrics.py
    test_network_metrics.py
    test_sentry.py
    test_tracing.py
    test_command_tracing.py
    test_pack_metrics.py
    test_profiling_bridge.py  # DROPPED — profiling bridge removed
    test_gauge_export.py
    bench_overhead.py

packages/maid-engine/tests/ai/
    test_completion_chunk.py
    test_streaming_fix.py

packages/maid-engine/tests/api/
    test_ai_pricing_api.py

deploy/prometheus/
    maid_alert_rules.yml.j2
    maid_alert_config.yml
    alertmanager_routing.yml

deploy/alertmanager/
    maid_routes.yml.j2

docs/dashboards/
    maid-server-health.json
    maid-ai-cost-overview.json

docs/runbooks/
    MaidTickLoopStalled.md
    MaidAIBudgetExhausted.md
    MaidTargetDown.md

docs/guides/
    operator-observability.md

Files Modified (Summary)¶

packages/maid-engine/pyproject.toml                              # Add structlog, prometheus-client deps + optional extras
packages/maid-engine/src/maid_engine/config/settings.py          # Add ObservabilitySettings model
packages/maid-engine/src/maid_engine/core/engine.py              # Create ObservabilityRegistry, start InternalServer, last_tick_monotonic
packages/maid-engine/src/maid_engine/core/events.py              # Add obs_registry to EventBus, event counter
packages/maid-engine/src/maid_engine/core/world.py               # Add O(1) entity counters
packages/maid-engine/src/maid_engine/commands/registry.py        # Add command instrumentation and tracing
packages/maid-engine/src/maid_engine/ai/providers/base.py        # LLMProvider refactor, CompletionChunk, TokenUsage
packages/maid-engine/src/maid_engine/ai/providers/anthropic.py   # Rename complete() to _do_complete()
packages/maid-engine/src/maid_engine/ai/providers/openai.py      # Rename complete() to _do_complete()
packages/maid-engine/src/maid_engine/ai/providers/ollama.py      # Rename complete() to _do_complete()
packages/maid-engine/src/maid_engine/ai/rate_limiter.py          # Add denial/allowed counters
packages/maid-engine/src/maid_engine/ai/circuit_breaker.py       # Add state gauge and trips counter
packages/maid-engine/src/maid_engine/api/admin/dashboard.py      # Unify MetricsCollector with Prometheus, add AIMetrics
packages/maid-engine/src/maid_engine/storage/document_store.py   # Add database operation metrics
packages/maid-engine/src/maid_engine/net/web/server.py           # Add network metrics, ASGI middleware

Dependency Graph¶

Phase 1 (Foundation):
  1.1 Package Structure ----+
  1.2 Settings ------------ | -+
  1.3 MaidContext --------- | -+-+
  1.4 Registry ----------- (1.2, 1.3) -+
  1.5 safe_observe ------- (1.4) -+
  1.6 structlog config --- (1.3) -+
  1.7 Privacy ------------ (1.2) -+
  1.8 Prometheus Metrics - (1.4) -+
  1.9 Log Sampling ------- (1.6) -+
  1.10 ScrapeCache ------- (1.8) -+
  1.11 HealthChecker ----- (1.4) -+
  1.12 Internal Server --- (1.10, 1.11) -+
  1.13 Tick Instrumentation (1.4, 1.5) -+
  1.14 Command Instrumentation (1.3, 1.4, 1.5)
  1.15 EventBus Instrumentation (1.4, 1.5)
  1.16 Entity Counting --- (none, parallel)
  1.17 GameEngine Integration (1.4, 1.11, 1.12, 1.13)
  1.18 ASGI Middleware --- (1.3, 1.4)
  1.19 Benchmark Gate ---- (1.13, 1.14, 1.15, 1.16) -- BLOCKER for Phase 2

Phase 2 (AI and Game Metrics):
  2.1 CompletionChunk ---- (none)
  2.2 LLMProvider Refactor (1.4, 2.1)
  2.3 Streaming Fix ------ (2.1)
  2.4 AI Cost Calculation - (2.1)
  2.5 Pricing Admin API -- (2.4)
  2.6 Rate Limiter Metrics (1.4)
  2.7 Circuit Breaker Metrics (1.4)
  2.8 MetricsCollector --- (1.8, 2.4)
  2.9 Database Metrics --- (1.4)
  2.10 Network Metrics --- (1.4)
  2.11 Sentry ------------ (1.4)

Phase 3 (Tracing, Dashboards, Polish):
  3.1 OTel Tracing ------- (1.4)
  3.2 Command/AI Tracing - (3.1, 1.14, 2.2)
  3.3 Grafana Dashboards - (1.8, 2.4)
  3.4 Pack Metric API ---- (1.4, 1.8)
  3.5 structlog Migration - (1.6)
  3.6 Profiling Bridge --- DROPPED
  3.7 Gauge Export Loop -- (1.16, 2.4)
  3.8 Client Error Ingestion DROPPED
  3.9 SLO/Alert Rules ---- (1.8)
  3.10 Runbooks ---------- (3.9)
  3.11 Operator Guide ---- (3.3, 3.9, 3.10)
  3.12 Alertmanager Template (3.9)

Success Criteria¶

[ ] All observability instrumentation is wrapped in safe_observe() and never crashes the game
[ ] Tick loop overhead with observability enabled is < 2% at 4 ticks/second (benchmark validated)
[ ] Command execution overhead is < 1ms additional latency per command
[ ] /metrics endpoint returns valid Prometheus text exposition format
[ ] /healthz, /readyz, /livez endpoints respond correctly on internal port 9090
[ ] structlog produces JSON output in production and console output in development
[ ] Player IDs are HMAC-anonymized in all log output when anonymize_player_ids is enabled
[ ] Non-whitelisted command arguments are redacted in log output
[ ] AdaptiveSampler always passes ERROR/CRITICAL/audit events through
[ ] CardinalityGuard prevents label explosion beyond configured limit
[ ] AI cost tracking accurately calculates USD cost from token usage
[ ] AICostTracker accumulates costs correctly across multiple providers and models
[ ] LLMProvider.complete() wrapper records duration, tokens, and request count
[ ] complete_streaming() returns CompletionChunk objects with cancellation safety
[ ] Circuit breaker state gauge reflects current state (CLOSED/OPEN/HALF_OPEN)
[ ] Rate limiter denial counter includes reason label
[ ] Database and network metrics are collected with correct labels
[ ] OpenTelemetry tracing is no-op when opentelemetry package is not installed
[ ] Sentry integration is no-op when sentry-sdk is not installed or DSN is empty
[ ] Content pack metrics are prefixed and label-validated
[ ] Grafana dashboard JSON files are importable
[ ] Alert rules cover tick loop, command latency, AI error rate, and AI budget SLOs
[ ] Runbooks exist for all critical alerts
[ ] NullObservabilityRegistry allows the engine to run with zero observability overhead
[ ] All new code has >80% test coverage
[ ] All public APIs have Google-style docstrings
[ ] MyPy strict mode passes on all new modules

Prerequisites / Blockers from Other Design Docs¶

No hard external blockers. The observability system is self-contained within maid-engine infrastructure.
Soft dependency on Doc 01 (Durable Persistence): AICostTracker can persist pricing configuration via DocumentStore. If Doc 01 has not landed, pricing falls back to file-based or environment variable configuration. The DocumentStore integration in load_pricing() is deferred to Phase 3 (task 2.5).
Soft dependency on Doc 02 (Database Migrations): Database metrics instrumentation hooks into PostgresDocumentStore. If the migration system changes DocumentStore internals, the instrumentation hooks in task 2.9 may need adjustment.
Integration with existing ProfileManager: The profiling bridge has been dropped from this plan. The existing ProfileManager at packages/maid-engine/src/maid_engine/profiling/ continues to operate independently; operators can correlate profiling sessions with Prometheus data via timestamps.
Integration with existing MetricsCollector: Task 2.8 refactors the admin dashboard MetricsCollector to read from Prometheus counters/gauges instead of re-computing. This is a non-breaking change; the admin API contract remains identical.
The existing AuditLogger at packages/maid-engine/src/maid_engine/logging/audit.py is not replaced. Audit logging and operational observability are complementary systems. The AdaptiveSampler (task 1.9) ensures audit-tagged events always pass through sampling.