Skip to content

Tier 2 AI Content Pipeline Review — R1

This review compares docs/designs/authoring/tier2-ai-content-pipeline.md against the current MAID codebase.

Verdict

The document is ambitious, but it currently overstates what the codebase can support. The biggest problem is that the design treats structured, schema-constrained generation as if it is already compatible with the existing LLM abstraction layer. It is not. The proposed CLI, in-game commands, budget/caching systems, and validation bridge are also described as if they are near-term evolutions of existing code, but most of the required modules do not exist yet.

1. BLOCKER — Structured output is the core of the design, but the current provider API cannot carry schemas, tool definitions, or response-format directives

Why this is bad

The document makes schema-constrained generation the central Tier 2 mechanism, but the actual LLMProvider API only supports plain text completions. There is nowhere to pass: - a JSON Schema - Anthropic tool definitions / tool choice - OpenAI response_format - Gemini response schema settings - a reproducibility seed - repair/validation metadata

Without widening the provider abstraction first, the “structured output engine” is not implementable as designed.

Design evidence - Provider-agnostic claim: tier2-ai-content-pipeline.md:98-110 - Structured adapter design: tier2-ai-content-pipeline.md:983-1200

Code evidence - CompletionOptions only has model, max_tokens, temperature, top_p, stop_sequences, presence_penalty, frequency_penalty: packages/maid-engine/src/maid_engine/ai/providers/base.py:91-102 - CompletionResult is just free-form content plus usage/metadata: packages/maid-engine/src/maid_engine/ai/providers/base.py:104-112 - LLMProvider.complete() just forwards those options: packages/maid-engine/src/maid_engine/ai/providers/base.py:177-221 - Anthropic provider uses plain messages.create(...) kwargs with no tools/schema: packages/maid-engine/src/maid_engine/ai/providers/anthropic.py:178-239 - OpenAI provider uses plain chat.completions.create(...) with no response_format: packages/maid-engine/src/maid_engine/ai/providers/openai.py:123-176 - Gemini provider does not expose response schemas either: packages/maid-engine/src/maid_engine/ai/providers/gemini.py:178-248 - Ollama provider only posts normal chat payloads: packages/maid-engine/src/maid_engine/ai/providers/ollama.py:131-187 - ChatJimmy provider is also plain chat: packages/maid-engine/src/maid_engine/ai/providers/chatjimmy.py:159-198

What the doc should do instead

State explicitly that Tier 2 starts with a breaking or additive expansion of the provider abstraction, and list the exact new fields the provider layer must support.

2. HIGH — The document describes a maid ai command group and in-game @ai commands that do not exist

Why this is bad

This is the most obvious trust failure in the document. It describes a polished command surface as if it is the natural interface today, but there is no maid ai command group at all, and no runtime @ai.* command registration file.

Design evidence - maid ai command group as primary interface: tier2-ai-content-pipeline.md:147-209 - In-game @ai command registration and examples: tier2-ai-content-pipeline.md:3372-3526 - Migration path claims maid dev generate is deprecated and forwards to maid ai generate: tier2-ai-content-pipeline.md:4290-4303

Code evidence - Top-level CLI registers no ai Typer app: packages/maid-engine/src/maid_engine/cli/app.py:28-100 - There is no packages/maid-engine/src/maid_engine/cli/ai.py - There is no packages/maid-engine/src/maid_engine/ai/commands/ai_commands.py - Current AI CLI is only maid dev generate: packages/maid-engine/src/maid_engine/cli/app.py:1175-1231 - maid --help shows no ai group; maid dev --help still exposes generate

Impact

Readers will assume most of Tier 2 already exists as scaffolding. It does not.

3. HIGH — maid dev generate is described as a migration target, but the actual command is a raw prompt stub, not a pipeline interface

Why this is bad

The document frames Tier 2 as an upgrade from the current command. The current command is not a weak version of Tier 2; it is a completely different shape.

Actual behavior today - Only supports room, item, npc - No provider override - No model override - No validation - No YAML serialization - No structured output - No retries / repair - No world context - No few-shot examples - No review workflow - No cost estimation

It prints whatever the provider returns and optionally writes it verbatim to disk.

Design evidence - Full CLI contract: tier2-ai-content-pipeline.md:166-209 - Deprecation/forwarding story: tier2-ai-content-pipeline.md:4292-4303

Code evidence - Current implementation: packages/maid-engine/src/maid_engine/cli/app.py:1175-1231

Extra problem

The system prompt literally asks for “JSON format”, but nothing validates JSON and nothing converts it to loader-compatible YAML.

4. HIGH — The “works without AI configured / use mock provider” story is factually wrong for the current settings defaults

Why this is bad

The document promises a graceful no-provider path and says mock/template generation can be used without API keys. In the actual codebase, the registry will usually register Ollama and ChatJimmy before it ever falls back to MockProvider.

That means a machine with no API keys is not “unconfigured” in practice: - ollama_host defaults to http://localhost:11434 - chatjimmy_enabled defaults to True - mock only appears when nothing else registers

So the default behavior is “try local Ollama, then maybe remote ChatJimmy,” not “warn and offer mock mode.”

Design evidence - Optional/no-provider behavior: tier2-ai-content-pipeline.md:4147-4168 - Local Ollama instructions: tier2-ai-content-pipeline.md:4191-4209

Code evidence - AI defaults: packages/maid-engine/src/maid_engine/config/settings.py:315-335 - default_provider = "anthropic" - ollama_host = "http://localhost:11434" - chatjimmy_enabled = True - Registry registration order and mock fallback: packages/maid-engine/src/maid_engine/ai/registry.py:293-380 - Registry tests confirm mock is only used when Ollama/ChatJimmy are disabled: packages/maid-engine/tests/ai/test_registry.py:318-372

Observed behavior

Running MAID_DEBUG=true uv run maid dev generate room 'Test Room' attempted a live POST to http://localhost:11434/api/chat and failed with ConnectError, which is the opposite of the document’s promised graceful fallback.

5. HIGH — The document ignores a real startup blocker: the current CLI may fail before any AI logic runs unless debug or a secure admin secret is configured

Why this is bad

The design treats CLI generation as always available, but in the current codebase settings construction can fail in non-debug mode because the default admin secret key is intentionally rejected. That means the current command is less reliable than the doc implies.

Code evidence - insecure default secret constant: packages/maid-engine/src/maid_engine/config/settings.py:66 - admin secret validation behavior: packages/maid-engine/src/maid_engine/config/settings.py:1148-1189 - get_settings() is called by dev_generate: packages/maid-engine/src/maid_engine/cli/app.py:1187-1192

Observed behavior

Running uv run maid dev generate ... without MAID_DEBUG=true failed during settings initialization due to the admin secret-key validation.

Why this matters to Tier 2

If Tier 2 is positioned as a CLI-first authoring system, the design needs to acknowledge existing boot requirements and how the AI commands will avoid or inherit them.

6. HIGH — Provider capability table is overstated and partially wrong

Why this is bad

The design’s provider matrix reads like a capability statement, not a future wish list. Right now it is materially inaccurate.

Design evidence - Capability matrix: tier2-ai-content-pipeline.md:100-110 - Adapter implementation sketch: tier2-ai-content-pipeline.md:1055-1200

Problems 1. Anthropic: doc assumes tool use with schema, but provider does not expose tools. 2. OpenAI: doc assumes response_format/JSON schema, but provider does not expose it. 3. Gemini: doc claims structured schema support, but adapter map does not even include Gemini (tier2-ai-content-pipeline.md:1193-1200). 4. Ollama / ChatJimmy: doc says “JSON mode + post-validation,” but current providers only do plain text chat calls.

Code evidence - Anthropic provider kwargs: packages/maid-engine/src/maid_engine/ai/providers/anthropic.py:178-196 - OpenAI provider kwargs: packages/maid-engine/src/maid_engine/ai/providers/openai.py:123-139 - Gemini provider config: packages/maid-engine/src/maid_engine/ai/providers/gemini.py:178-205 - Ollama payload: packages/maid-engine/src/maid_engine/ai/providers/ollama.py:131-145 - ChatJimmy request path: packages/maid-engine/src/maid_engine/ai/providers/chatjimmy.py:159-169

7. HIGH — The document’s “standardized output format” drifts from actual component models and fixture formats

Why this is bad

This is exactly the kind of subtle schema drift that will poison prompts, examples, and agent instructions.

Design evidence - Standardized output requirements for agents: tier2-ai-content-pipeline.md:2476-2500 - It claims examples such as: - NPCComponent: npc_type, behavior, faction, level, spawn_room - ItemComponent: item_type, weight, base_value

Code evidence - Actual NPCComponent fields are template_id, behavior_type, dialogue_id, spawn_point_id, respawn_time, wander_radius, faction_id, is_merchant, is_quest_giver: packages/maid-stdlib/src/maid_stdlib/components/core.py:422-438 - Actual ItemComponent uses value, not base_value: packages/maid-stdlib/src/maid_stdlib/components/core.py:440-463 - Actual valid NPC fixture uses behavior_type and a top-level location, not spawn_room: packages/maid-engine/tests/fixtures/data/valid/npcs.yaml:1-13

Impact

The doc is teaching agents and humans the wrong field names.

8. HIGH — Budgeting and caching are specified as if there is an existing foundation, but the current code only has dialogue budgets and a memory-context cache

Why this is bad

The design repeatedly implies it is extending existing infrastructure. In practice it is introducing an entirely new budget/caching stack.

Design evidence - Cost/budget sections: tier2-ai-content-pipeline.md:3226-3325, 3820-4099 - maid ai cache and @ai budget: tier2-ai-content-pipeline.md:163, 3514-3526

Code evidence - MemoryCache is only for memory context sections, not generation responses: packages/maid-engine/src/maid_engine/ai/cache.py:1-69 - Existing RateLimiter only handles RPM and daily token budgets for dialogue: packages/maid-engine/src/maid_engine/ai/rate_limiter.py:61-76 - Existing settings only define AISettings and AIDialogueSettings; there is no AIGenerationSettings: packages/maid-engine/src/maid_engine/config/settings.py:315-398 - Existing token-budget manager is for context-provider allocation, not request/cost governance: packages/maid-engine/src/maid_engine/ai/token_budget.py:1-82

Missed opportunity

The design also hardcodes a new pricing table instead of integrating with the existing observability pricing/cost helpers (packages/maid-engine/src/maid_engine/observability/ai_metrics.py:23-155). That is a guaranteed source-of-truth split.

9. MEDIUM — The doc says all generation uses LLMProviderRegistry.complete_with_fallback(), but its own sketches bypass the registry

Why this is bad

The design is internally inconsistent.

Design evidence - Claim: tier2-ai-content-pipeline.md:100 - Adapter sketch directly calling a single provider: tier2-ai-content-pipeline.md:1080-1083, 1124-1125, 1156-1158

Code evidence - Registry fallback API exists: packages/maid-engine/src/maid_engine/ai/registry.py:184-256 - NPC dialogue does not use it; it hand-rolls a fallback chain and then calls the provider directly: packages/maid-classic-rpg/src/maid_classic_rpg/systems/npc/dialogue.py:600-716

Why this matters

If Tier 2 wants consistent fallback, cost tracking, and provider selection, the design needs one authoritative path. Right now it documents two.

10. MEDIUM — Batch generation is presented as parallel, but the sample implementation is sequential

Why this is bad

The design claims realistic batch generation with concurrency control, but the sample code processes items one at a time.

Design evidence - “Parallelism and Rate Limiting”: tier2-ai-content-pipeline.md:3327-3367

Problem in the sample

The loop does: 

for item in items:
    async with self._semaphore:
        result = await self._generate_single(item)
That is still serial. No tasks are created. The semaphore just limits a concurrency level that never exceeds 1.

Impact

The doc’s batch cost/time assumptions are optimistic because the shown architecture does not actually parallelize work.

11. MEDIUM — Several commands that the design depends on have no proper CLI contract

Why this is bad

The document is selective about which commands get full I/O specs.

Missing or underspecified surfaces - maid ai cache [stats|clear|export] appears in the overview but has no detailed section: tier2-ai-content-pipeline.md:163 - maid ai validate-output is central to agent integration but has no formal CLI spec: tier2-ai-content-pipeline.md:2453-2500 - maid ai schema is required for agent template injection but also lacks a contract: tier2-ai-content-pipeline.md:2535-2543 - In-game examples reference @ai.accept, @ai.reject, @ai.edit, and review flows, but only a subset of @ai commands are specified: tier2-ai-content-pipeline.md:3411-3439, 3472, 3511

Impact

These are not minor helper commands. They are part of the workflow-critical path, yet their arguments, outputs, and failure semantics are undefined.

12. MEDIUM — Custom prompt templates per content pack are not aligned with the actual ContentPack protocol

Why this is bad

The doc proposes pack-level prompt overrides as if there is already a hook for them. There is not.

Design evidence - Proposed get_ai_prompt_templates() extension: tier2-ai-content-pipeline.md:4115-4145

Code evidence - Actual ContentPack protocol methods: packages/maid-engine/src/maid_engine/plugins/protocol.py:31-216 - No AI-template hook exists there

Impact

This is a real extension-point change, not a documentation tweak. The design should treat it as a protocol evolution with migration implications for packs, not as a free add-on.

13. MEDIUM — Security/safety coverage is too thin for a system that ingests world state, context files, and agent-generated YAML

Why this is bad

The document discusses budgets and malformed output, but it under-specifies the actual security model for generation.

Gaps 1. Prompt injection - maid ai generate accepts repeatable --context PATH and optional world-context injection: tier2-ai-content-pipeline.md:193-199 - World-aware generation is a major theme: tier2-ai-content-pipeline.md:2547-2797 - There is no concrete trust-boundary strategy for hostile or poisoned context files/YAML. 2. PII - The codebase already has a PIIRedactor: packages/maid-engine/src/maid_engine/ai/pii.py:1-70 - The design never mentions redacting PII from prompts, world extracts, logs, or generated review output. 3. Generated-content filtering - The codebase has ContentFilter, but it is currently geared to dialogue safety: packages/maid-engine/src/maid_engine/ai/safety.py:1-31 - The design does not specify if generated lore/NPCs/items/descriptions are safety-screened before being written.

Impact

The current security story is basically “validate schema and budget tokens,” which is nowhere near enough for automated content synthesis.

14. MEDIUM — Determinism/reproducibility is promised, but the current API offers no support

Why this is bad

The design exposes --seed INT and talks as if reproducible generation is part of the experience.

Design evidence - --seed INT option: tier2-ai-content-pipeline.md:203-206

Code evidence - No seed field in CompletionOptions: packages/maid-engine/src/maid_engine/ai/providers/base.py:91-102 - No seed plumbing in any provider implementation - MockProvider is sequence-based, not seed-based: packages/maid-engine/src/maid_engine/ai/providers/base.py:486-538 - Current CLI hardcodes temperature=0.8: packages/maid-engine/src/maid_engine/cli/app.py:1212-1221

Impact

Without a real determinism contract, the seed flag is just UX theater.

15. MEDIUM — “Local LLM only” mode is not actually nailed down

Why this is bad

The document claims Ollama gives local/no-cost generation with an identical interface. That is true only if the runtime is prevented from falling back to remote providers.

Design evidence - Ollama section: tier2-ai-content-pipeline.md:4191-4209

Code evidence - chatjimmy_enabled defaults to True: packages/maid-engine/src/maid_engine/config/settings.py:329-331 - Registry registration includes ChatJimmy by default when enabled: packages/maid-engine/src/maid_engine/ai/registry.py:359-370

Problem

The doc tells users to set MAID_AI_DEFAULT_PROVIDER=ollama, but does not tell them to disable ChatJimmy. If Tier 2 uses registry fallback, a supposedly local-only workflow may still make remote calls.

16. LOW — Token accounting in the structured-output sketch is subtly wrong for current CompletionResult usage

Why this is bad

The design’s StructuredResult.tokens_used returns usage.get("total_tokens", 0).

Design evidence - StructuredResult.tokens_used: tier2-ai-content-pipeline.md:1049-1052

Code evidence - Many providers populate only prompt_tokens and completion_tokens, not total_tokens: - Anthropic: packages/maid-engine/src/maid_engine/ai/providers/anthropic.py:219-229 - OpenAI: packages/maid-engine/src/maid_engine/ai/providers/openai.py:161-166 - Ollama: packages/maid-engine/src/maid_engine/ai/providers/ollama.py:171-176

Impact

The proposed tokens_used property would undercount or report zero for common successful responses unless extra normalization is added.

17. LOW — Tier 1 dependency is acknowledged, but the contract with Tier 1 is still underspecified

Why this is bad

The document says Tier 2 depends on Tier 1, but it still handwaves the exact integration seam.

Design evidence - dependency banner: tier2-ai-content-pipeline.md:6 - repeated “run Tier 1 validation” language: tier2-ai-content-pipeline.md:95-96, 130, 199-200

Code evidence - Actual loader pipeline is six phases (Discover → Parse → Prepare → ResolveRefs → Instantiate → PostLoad): packages/maid-engine/src/maid_engine/loader/pipeline.py:179-194 - PreparePhase is real, but not the whole story for reference resolution or instantiation

Concern

The doc often talks as if “Tier 1 validation” is a single reusable step. In reality, some guarantees only emerge after later loader phases. That matters for claims like cross-reference validation and “ready for load.”

Bottom line

The design is strongest as a vision document and weakest as an implementation-guiding spec. The biggest fixes needed before this should guide engineering work are:

  1. Redesign the provider abstraction for structured generation.
  2. Separate “future commands” from “current commands” more honestly.
  3. Reconcile default provider behavior, offline behavior, and mock behavior with actual settings/registry code.
  4. Stop teaching wrong schema fields in examples.
  5. Define the security model for world-context ingestion, prompt injection, PII, and generated-content filtering.
  6. Replace the fictional migration path with an explicit phased rollout plan.