Skip to content

ADR-008: Scenario Labels as Concrete Actions

Status: Accepted Date: 2026-03-07 Context: Early AI-generated scenario options used abstract virtue-signaling labels ("Uphold ethical principles," "Transparent disclosure," "Team consensus building"). These labels exposed the underlying driver classification system and made one option obviously "right."

Decision

Option labels must name the concrete action the player would take. The scenario generation prompt includes:

Banned patterns: - Abstract virtues: "Uphold ethical principles", "Fair values-based approach" - Driver keywords as labels: "Transparent disclosure", "Team consensus building" - Vague process: "Practical compromise solution", "Bold growth investment"

Required patterns: - Specific mechanisms: "Implement quadratic voting" - Concrete numbers: "Cut 3 underperforming projects" - Actual tradeoffs: "Split budget 60/40, defer hiring" - Specific timelines: "6-month trial with monthly reviews"

Option descriptions must explain practical consequences and tradeoffs, not restate the value behind the choice.

Rationale

Scenarios that signal "the right answer" don't produce useful data. If an option says "prioritize team wellbeing" vs "ignore concerns," nobody picks the second one. The resulting data tells us nothing about the player's actual values — only that they can identify the socially desirable answer.

Concrete action labels force genuine tradeoff reasoning. "Cut 3 underperforming projects" vs "6-month trial with monthly reviews" both sound reasonable — the choice reveals whether the player values decisive action or measured evaluation.

This also prevents gaming: if players can see which option maps to which driver (e.g., "Team consensus building" is obviously team_harmony), they can manipulate their profile.

Alternatives Considered

  • Post-hoc label rewriting: Generate with abstract labels, then rewrite. Rejected because the abstract framing influences the entire scenario structure, not just the labels.
  • Hiding labels (show only descriptions): Considered but labels serve as cognitive anchors that help players compare options quickly.
  • Randomized driver-label mapping: Make the mapping unpredictable. Rejected because the real problem is that abstract labels produce low-information choices, not just that the mapping is visible.

Consequences

  • Scenario generation takes slightly more prompt tokens due to the banned/required patterns list.
  • Existing scenarios with abstract labels were manually rewritten (11 scenarios fixed in Week 7).
  • The ban list in the prompt must be maintained as new anti-patterns emerge.

How We Got Here

The first batch of AI-generated scenarios looked great on paper but produced terrible data. Option labels like "Uphold ethical principles" vs "Maximize operational efficiency" vs "Build team consensus" vs "Take calculated risk" weren't real decisions — they were a personality quiz with obvious answers.

The signal this produced was almost entirely social desirability bias. Players chose whichever option sounded most "leader-like." The driver classifications were technically correct (the player did pick the "principle" option) but revealed nothing about their actual decision-making — just that they could read the labels.

The fix came in two parts:

Part 1: Recognizing the failure mode. The scenario quality pass in Week 7 audited all generated scenarios and found a pattern: Claude's natural tendency when generating organizational dilemmas is to frame options as value archetypes ("the principled approach" vs "the pragmatic approach"). This is how business school case studies work — but case studies are designed to teach, not to measure. For measurement, you need choices that all sound reasonable.

Part 2: Prompt engineering the solution. Rather than trying to fix labels after generation, the prompt was redesigned to prevent the failure mode. Banned patterns block the most common abstract framings. Required patterns force specificity (dollar amounts, headcounts, timelines, specific actions). The key test: "Would this label make sense as a menu item in a decision-making app? If it sounds like a fortune cookie or a leadership seminar title, rewrite it."

This also addressed a gaming concern: if players can see which option maps to which driver (e.g., "Team consensus building" is obviously team_harmony), sophisticated players could manipulate their profile toward whatever pattern they think the DTA rewards. Concrete action labels make the driver mapping opaque without hiding any information.

The option description rules were added later for the same reason — descriptions that said "Based on core values and principles" or "Build team unity" were restating the driver in different words, making the mapping obvious even when the label was concrete.

Key files: - src/lib/ai/scenario-generator.ts — option label rules in the generation prompt (lines 96-128)