ADR-061: Scenario generation quality — pre-commitment planning block

Status: Accepted Date: 2026-05-18 Context: AI-generated scenarios for multiplayer quick rounds and campaigns were coming out thin — no named characters, no prior history, no concrete stakes, no per-option lock-in costs. The generation prompt already contained a detailed checklist of required elements (named characters with quotes, prior history with time reference, worst-case with real numbers, lock-in mechanism, no escape hatches without stated failure costs), but the model was satisfying the checklist superficially while still producing generic, characterless scenarios.

The underlying problem: by the time the model reaches a checklist-as-verification block at the end of a long prompt, it has already written the description mentally and uses the checklist to rationalize compliance rather than to plan. LLMs tend to commit to their first draft and then satisfy constraints post-hoc.

At a glance

What it decides: Scenario generation now forces a planning block as the mandatory first field in the JSON response — the model commits to named characters, prior history, worst-case, and per-option lock-in before writing the description, which must then draw from those committed values.

• Pre-commitment beats post-hoc verification — a checklist at the end of a long prompt gets rationalized; structured data first changes the generation order, not just the constraint.
• lockInPerOption was the hardest gap — it forces an explicit failure cost for each option (including hedges like "do both"/"request more info") before that option is written.
• Narrative voice removed from scenario-generator.ts — case-study bodies read like case studies; cinematic Hook+Briefing framing belongs only in campaign preambles (campaign-narrator.ts). maxTokens 2048 → 2500.
• Rejected a separate pre-flight validation call (doubles cost/latency) and per-theme prompts (would duplicate planning logic six times).
• Watch for: truncated options arrays if a scenario hits the 2500 ceiling; planning block isn't stored or shown — it's the right source if corpus-quality auditing becomes a priority.

flowchart LR
    A["Long generation prompt<br/>(checklist at end)"] -->|old: write first,<br/>verify after| B["Generic, characterless<br/>scenario"]
    C["planning block<br/>(first JSON field)"] -->|"namedCharacters,<br/>priorHistory, worstCase,<br/>lockInPerOption"| D["Description drawn from<br/>committed values"]
    D --> E["Concrete scenario:<br/>named cast, real numbers,<br/>per-option lock-in"]

Pre-commitment reorders generation: commit the structured values, then write the body from them.

Decision

Insert a planning block as a mandatory first field in the JSON response, forcing the model to commit to specific pre-populated values before writing the scenario description. The description must then draw from the planning block.

The planning block requires:

"planning": {
  "namedCharacters": [
    "Name (role): their explicitly stated position or direct quote",
    "Name (role): their explicitly stated position or direct quote"
  ],
  "priorHistory": "A specific past event with time reference",
  "worstCase": "Specific worst-case in plain human terms — names, numbers, what happens to real people",
  "lockInMechanism": "What makes this decision irreversible",
  "lockInPerOption": {
    "A": "Specific thing burned/foreclosed by picking A",
    "B": "Specific thing burned/foreclosed by picking B",
    "C": "If hedge: what fails or who objects if it doesn't hold",
    "D": "Specific thing burned/foreclosed by picking D"
  }
}

The checklist before JSON Response: is retained but reframed from "verify" to "populate then verify" — the model is instructed to fill out the planning block first, then write the description using those committed elements.

Additionally, narrative voice (Hook+Briefing literary structure) was removed entirely from scenario-generator.ts. Campaign preambles already handle narrative framing via campaign-narrator.ts; the scenario's case-study body should read like a case study, not a short story.

maxTokens bumped 2048 → 2500 to accommodate the planning block without compressing the scenario body.

Rationale

Pre-commitment beats post-hoc verification. The planning block forces the model to name characters, surface prior history, and articulate per-option lock-in before the narrative starts. It cannot satisfice later — it has already committed to "Calloway (supply chief): 'That relay is the only way we know Creekside is still alive'" and must deliver Calloway with that quote in the description.

Per-option lock-in was the hardest gap to close. Options like "do both at half speed" or "request more information" were passing the general checklist because they technically had a stated failure cost somewhere in the description — but it was vague. lockInPerOption forces the model to commit "C: If you try to run both tracks, Vasquez will pull her team off the survey entirely" before it writes Option C, making the hedge cost unavoidable.

Narrative voice in case studies was an architecture mistake. The narrativeVoice: true flag on campaign arc theme_config was flowing into buildScenarioGenerationPrompt, wrapping campaign case studies in cinematic Hook+Briefing prose. That structure is appropriate for preambles (scene-setting before a chapter), not for the decision-substrate body (which should be concrete and clinical). The fix is architectural: narrative voice belongs in campaign-narrator.ts only.

Alternatives Considered

• Pre-flight validation gate (separate API call): Generate the scenario, then run a second LLM call to check quality and regenerate if it fails. Rejected because it doubles cost on every generation, adds latency, and is harder to tune. The root problem is prompt design, not output validation.
• Stronger checklist with explicit failure examples: Add examples of bad output to the checklist ("NOT 'a colleague' — must be 'Dr. Vasquez (head of facilities)'"). This was tried in earlier iterations and helped at the margins but didn't eliminate the problem because the model can still write the description and rationalize afterwards. Pre-commitment changes the generation order, not just the constraint.
• One prompt per theme: Rejected — the core quality problem (thin characters, no history, no lock-in) is theme-agnostic. Theme-specific prompts would duplicate the planning block logic six times and drift out of sync. The custom-description override already handles theme-specific world constraints.

Discussion

Jonathan first surfaced the problem with The Harvest Split (a multiplayer quick-round scenario): "There is no case study, barely any details." The prompt at that point had a strong-looking checklist, but the checklist was formatted as a verification gate — "before responding, verify each of the following" — which doesn't change what the model writes first.

The insight that resolved this was understanding that LLM attention degrades over long prompts: by line 200, a checklist isn't driving generation, it's being pattern-matched. Pre-commitment solves this by making the structured data come first in the JSON response, before the free-form description field.

The lockInPerOption field was added in a second tightening pass after testing showed scenarios where C/D options were hedges without explicit failure conditions. Jonathan asked to "tighten" and specifically approved adding per-option lock-in.

Narrative voice removal came out of a separate discovery: Jonathan noticed some test prompts had a "Hook/Briefing" block. Investigation revealed narrativeVoice: true was hardcoded in campaign arc creation and was flowing into scenario generation rather than being isolated to preambles. Jonathan clarified the architecture: "Quick rounds we shouldn't have the option at all. And for campaigns it is the default with the preamble (but not in the actual case study — case study part is like a quick round)."

All 6 themes tested via copy-paste into fresh Claude.ai conversations to validate the prompt without burning API credits (per feedback from a prior session). All passed: 2+ named characters with quotes, prior history with time reference, specific worst-case, hard deadline, and per-option lock-in costs present in every scenario generated.

Consequences

• All newly generated scenarios across all themes should have named characters, prior history, and per-option lock-in by default. The planning block is visible in the JSON response and can be inspected for quality auditing.
• Campaign case studies no longer get literary framing — they read like case studies. Campaign chapters still get narrative framing via the preamble (unchanged).
• If the model truncates output at maxTokens, the scenario body may be cut short. The 2500 limit should be sufficient; watch for truncated options arrays if very long scenarios are generated.
• The planning block is returned in the raw JSON response. It is not currently stored in the database or shown to players. If corpus quality analysis becomes a priority, the planning block is the right source — it surfaces what the model committed to before writing.
• Recursive quality tracking (logging planning-block adherence per generated scenario) was discussed but deferred. Ship and observe first.

Key files: - src/lib/ai/scenario-generator.ts — planning block in JSON response format, pre-commitment checklist, narrative voice block removed, maxTokens 2048 → 2500 - src/lib/ai/scenario-themes.ts — narrativeVoice?: boolean removed from ScenarioThemeConfig - src/types/database.ts — narrativeVoice?: boolean removed from ThemeConfig - src/components/game/ThemeSelector.tsx — narrative voice toggle props removed - src/app/solo/page.tsx — narrative voice state and handlers removed - src/app/play/page.tsx — narrative voice state removed - src/app/api/sessions/start/route.ts — narrative voice removed from scenario generation context - src/app/api/campaigns/route.ts — theme_config: { narrativeVoice: true } → theme_config: {} - src/app/api/campaigns/[id]/ready/route.ts — narrative voice fallback removed - src/app/api/campaigns/[id]/preamble/route.ts — narrativeVoice: false removed from campaign theme config - scripts/generate-test-prompts.ts — new; generates 6 copy-paste test prompts for manual theme testing