ADR-043: Per-option driver + scenario trigger tagging¶

Status: Accepted (implemented May 6, 2026) Date: 2026-05-06 Context: Cici's May 6 design audit identified the largest single data gap in Sync: the scenarios.options JSON shape stored only {id, label, description} per option, with no semantic tag for which of the 8 reasoning drivers each option represented. Similarly, the scenario row had no trigger column — the production code was hardcoding 'competitive_pressure' as a fallback in surfaces that needed one. Without these tags, three of the redesign's screens (Round, Reveal, Recent) render degraded data, the Insights "Triggers that activate this driver" surface can't render at all, and contextual analysis (the whole point of context_rules on player_decision_profiles) is context-blind. This ADR captures the decision to fix both at once.

At a glance

What it decides: Every scenario option is tagged with one of 8 drivers and every scenario with one of 10 triggers, so the system records what each choice represented (the signal), not just which option was picked. New scenarios get tagged at generation; the existing corpus was backfilled.

Tagged at generation time, not post-hoc — the generator emits driver + trigger inline (the model is already reasoning about the scenario, so the tag is near-free and can't drift from intent); the validator rejects responses missing either field.
Rejected: rule-based / keyword tagging — drivers are conceptual, not lexical ("expedient" vs "pragmatic" vs "efficient" cluster together); the rationale-keyword trigger detector in player-model.ts is a complementary signal (what context the player invoked), not a substitute for the scenario's own tag.
Backfill model: Sonnet 4.6, not Haiku 4.5 — Haiku failed the 85% ground-truth bar (over-picked crisis_mentioned; weak on the pragmatism↔caution / pragmatism↔efficiency boundaries). Cost rose to ~$3–5 once, still trivial.
Known ceiling: ~85–90% inter-rater agreement on adjacent driver pairs is a property of the taxonomy, not a tagger bug. Watch: don't feed borderline tags into high-stakes downstream inference without flagging low agreement. Generation now depends on migration 087.

flowchart TD
    G["Scenario generator v1.5.0"] --> O["Per option: pick 1 of 8 drivers"]
    G --> TR["Per scenario: pick 1 of 10 triggers"]
    O --> V{"Validator:<br/>fields present + valid enum?"}
    TR --> V
    V -->|"no"| RG["Regenerate retry"]
    RG --> G
    V -->|"yes"| DB[("scenarios row + options[].driver")]
    BF["Backfill (Sonnet 4.6)"] --> DB
    DB --> SC["Round / Reveal / Recent chips"]
    DB --> IN["Insights: triggers-per-driver"]
    DB --> CR["context_rules contextual analysis"]

Forward path (generator + validator) and backward path (one-time Sonnet backfill) both populate the same tags, which unblock three screens, the Insights trigger surface, and context-conditioned analysis.

Decision¶

Tag every scenario option with one of 8 drivers (team_harmony, efficiency, principle, pragmatism, caution, growth, transparency, autonomy) and every scenario with one of 10 triggers (crisis_mentioned, team_conflict, resource_constraint, opportunity, ethical_dilemma, external_pressure, competitive_pressure, innovation_challenge, authority_threat, relational_proximity).

Two execution paths, in parallel:

Forward — scenario generator emits both fields. The generation prompt (src/lib/ai/scenario-generator.ts, version bumped to v1.5.0) teaches the model the 8 drivers and 10 triggers and requires one driver per option + one trigger per scenario. The validator rejects responses missing either field or with invalid enum values, triggering the existing regenerate retry. New scenarios always have these tags from the moment they're created.
Backward — Haiku 4.5 backfills the existing corpus. A one-time pass via scripts/backfill-scenario-tags.ts classifies the existing 895 scenarios (3,580 options total) using Haiku 4.5. Resumable, dry-run by default, ~10–20 minutes wall-clock at concurrency 3, ~$1–2 in API cost.

Schema (migration 087): - scenarios.trigger TEXT NULL with CHECK constraint listing the 10 valid values + index for the future by-trigger filter - scenarios.options[].driver — JSON-shape addition (no migration needed; existing reading code tolerates extra fields)

Both fields are nullable on legacy rows so existing scenarios keep working unchanged until the backfill runs.

Rationale¶

The drivers and triggers are what make Sync a profile-builder rather than a glorified poll. Without them, the system records "Jonathan picked option B" but doesn't know "option B was the pragmatic choice over the principled one" — and that distinction is the signal. Three concrete consequences:

Round/Reveal/Recent screens can render colored chips per option (visual cue of what each represents), the "your twin predicted you'd pick principle, but you went pragmatism" callout on Reveal, and dot patterns on Recent showing your driver pattern over time.
Insights "Triggers that activate this driver" surface is literally unrenderable without scenario.trigger populated. The eyebrow on Round ("trigger / competitive_pressure") was hardcoded specifically because there was no real value to read.
Contextual analysis in context_rules only works when scenarios are tagged with their context. Without it, "Jonathan is principled UNTIL there's a crisis, then he flips to pragmatism" — the whole point of contextual measurement — collapses to "Jonathan is principled (sometimes)."

The cost is negligible. Backfill is one-time ~$1–2 on Haiku 4.5 (cheap classifier, sufficient for picking from a 10-element enum with full scenario context). Per-new-scenario marginal cost is the 30-or-so output tokens of the driver + trigger fields against an existing Sonnet 4.6 generation that already costs ~$0.03 — call it a 1–2% adder.

Generate-time tagging beats post-hoc tagging because it's free (the model is already reasoning about the scenario; one extra structured field costs almost nothing) and it can't drift from the scenario's intent (the same model that wrote the option labels picks the driver that fits them).

Alternatives Considered¶

Manual classification by the team. Rejected. 3,580 options at ~30 seconds each is ~30 hours of grunt work, and humans are worse than an LLM at consistent-rubric classification at this volume — accuracy drifts after the first 200 calls and re-rubrication takes longer than just re-running the LLM. False economy.
Sonnet 4.6 for the backfill instead of Haiku 4.5. Rejected. ~3–5× the cost (~$3–5 vs ~$1–2) for marginal accuracy gain on a classification task where Haiku is plenty. The validator's enum constraint catches obvious mis-tags; spot-checking a sample of 30 taggings is faster than paying 5× to slightly improve the baseline.
Post-hoc rule-based tagger (regex/keyword over option text). Rejected. Drivers are conceptual, not lexical — "expedient" vs "pragmatic" vs "efficient" all map to similar driver clusters but a keyword tagger can't distinguish them from context. Plus the context-trigger detection in src/lib/ai/player-model.ts already does keyword-style trigger inference at decision time on the rationale text — that's a parallel signal, not a replacement for the scenario's own trigger tag.
Defer until the screens are actually being built. Rejected. The screens are being designed now; data has to land first or every Cici design review hits the same blocker. This is the single biggest unblock in the audit.
A separate scenario_option_drivers table instead of a JSON-shape addition. Rejected. Adds a join to every scenario read for one extra string per option. The JSON additive approach is forward-compatible with reading code, doesn't require any schema migration for the option side, and the data is conceptually owned by the option (not separately referenced). Per Cici's audit: "(or a new scenario_option_drivers table)" — the audit allowed both, JSON wins on simplicity.
Skip A.3 (scenarios.prompt column). Was Cici's recommendation given the default works fine. Overruled per the user's record-it-now principle: the column is nullable, costs nothing to have, and avoids a follow-up migration when scenarios eventually want varied phrasing. Cost-of-keeping ≈ 0; cost-of-adding-later > 0.

Discussion¶

The audit was clear that A.1 (per-option driver) was the single biggest gap — three screens degrade without it. A.2 (scenario trigger) is the obvious companion: same kind of structured tag, same kind of LLM-generated classification, same backfill pass. Bundling them is the right move because the cost of an LLM call is dominated by the prompt context (which is identical for both classifications) — getting the trigger "for free" with the same call that picks drivers is meaningfully cheaper than two separate passes.

The decision not to do post-hoc rule-based tagging deserves a callout. The system already has TRIGGER_KEYWORDS in src/lib/ai/player-model.ts that does keyword-based context detection from the rationale text the player wrote at decision time. That's a different signal: "what context did the player invoke when they explained their choice?" The scenario's own trigger tag is "what context did the scenario present?" Both matter; they're complements, not substitutes. The scenario tag is needed to support surfaces that show before the player decides (which trigger frame is this scenario operating under?), and to anchor context_rules analyses without circular dependence on the rationale text.

The risk worth watching: tagging quality. The validator catches obviously invalid tags (wrong enum values, missing fields), but it can't catch subtly-wrong tags ("transparency" when "principle" was more apt). Mitigation is a manual spot-check of ~30 random taggings after the backfill runs. If accuracy looks off, the backfill is idempotent — re-run after iterating on the prompt and the same scenarios get re-tagged with the improved version. This is one of the reasons backfill is --dry-run by default rather than --apply.

A subtle implementation choice: the validator now rejects scenarios without driver/trigger fields. That means scenario generation depends on migration 087 being applied (otherwise the insert fails on the missing trigger column). If 087 isn't applied, generation breaks. Acceptable because the migration is small and additive; mentioning here so future archeology understands the coupling.

Consequences¶

Cost paid once: ~$1–2 on Haiku for backfill of 895 scenarios. After that, per-new-scenario marginal cost is ~1–2% addition to existing Sonnet generation.
Three Cici screens unblock simultaneously (Round, Reveal, Recent) once 087 is applied + backfill runs.
Insights "Triggers that activate this driver" surface becomes renderable with real data.
Contextual analysis in context_rules gains a properly-tagged anchor; previously every scenario contributed to context-trigger inference based only on rationale-text keywords.
The hardcoded 'competitive_pressure' fallback in surfaces that need a trigger value can be retired once backfill runs and the column is universally populated.
Generation depends on migration 087. If 087 isn't applied, scenario generation breaks (insert fails on missing column). The migration is small, additive, and reversible — apply it before relying on the new generator path.
Watch: tagging quality on the backfill output. Spot-check 30 random scenarios after applying. If subtle mis-tags show up systematically (e.g. always confusing transparency with principle), iterate the backfill prompt and re-run — the script is idempotent and overwrites prior taggings.
Watch: new-scenario validation failures. The validator now rejects responses missing driver/trigger; the existing regenerate retry handles transient failures, but if certain scenario themes systematically fail to emit valid tags, the prompt needs adjustment for those themes.

Key files: - migrations/087_cici_audit_columns.sql — scenarios.trigger (with CHECK + index), scenarios.prompt, results.sync_score_after - src/lib/ai/driver-trigger-constants.ts — single source of truth for the 8 drivers + 10 triggers + branded DeclaredDriver / ExhibitedDriver types - src/lib/ai/scenario-tagger.ts — shared tagger used by backfill and eval; production model is Sonnet 4.6 after the May 6 eval - src/lib/ai/scenario-generator.ts — v1.5.0; prompt section "Driver and Trigger Tagging (REQUIRED)"; validator updates; toDbScenario passes trigger through - src/types/database.ts — ScenarioOption.driver: DeclaredDriver, Choice.classified_driver: ExhibitedDriver, Scenario.trigger/.prompt - scripts/backfill-scenario-tags.ts — Sonnet 4.6 pass; --dry-run default, --apply to write, --concurrency tunable, resumable - scripts/eval-scenario-tags.ts — --prepare / --suggest / --evaluate modes for ground-truth quality validation

Tier 1 ground-truth eval results (May 6, 2026)¶

After Tier 1 mitigations were in place, ran the ground-truth eval on 20 random scenarios. Sonnet 4.6 generated proposed tags via --suggest; Opus 4.7 spot-checked all 20 manually as a third data point; Haiku 4.5 was evaluated against the Sonnet+Opus baseline.

Result: Haiku 4.5 failed the 85% bar.

Model	Trigger agreement	Driver agreement
Haiku 4.5 vs Sonnet 4.6	50%	61%
Opus 4.7 vs Sonnet 4.6 (manual spot-check)	~85%	~88%

Haiku exhibited two systematic failure modes:

Over-picks crisis_mentioned — labels anything with urgency framing as crisis even when more specific triggers (resource_constraint, external_pressure, ethical_dilemma) are clearly the primary signal. 5 of the 10 trigger mismatches were Haiku picking crisis_mentioned over a more specific trigger.
Aggressive swaps in the pragmatism ↔ caution and pragmatism ↔ efficiency boundaries — Haiku's discrimination between these adjacent drivers is noticeably worse than Sonnet's, accounting for ~60% of the driver mismatches.

The Opus spot-check separately confirmed Sonnet's output is at the quality bar a careful human would set; the Opus/Sonnet disagreements cluster on genuine taxonomy edge cases (see "Taxonomy fuzziness" below), not on errors.

Decision: switch production tagger from Haiku 4.5 to Sonnet 4.6. Cost rises from ~$1–2 to ~$3–5 for the full 895-scenario backfill; still trivial at any scale. Per-new-scenario marginal cost rises from ~1% to ~3–5% of an already-cheap Sonnet generation call.

Known limitation: taxonomy fuzziness on adjacent drivers¶

The Opus spot-check surfaced a structural property of the 8-driver taxonomy: certain driver pairs (notably pragmatism ↔ caution and pragmatism ↔ efficiency) have legitimately overlapping conceptual territory. On options that combine "find a workable middle path" (pragmatism) with "minimize downside risk" (caution), even careful human raters will split. This is not a tagger problem — it's a property of the taxonomy.

Two implications:

A perfect 100% inter-rater agreement is unattainable without sharper definitions or a different categorical structure. Empirical ceiling is probably ~85–90% on these adjacent pairs.
Borderline tags should not feed downstream high-stakes inference. Where the pragmatism/caution call is genuinely ambiguous, treating the tag as ground truth in trust-graph or prediction analytics over-claims the data's certainty. This argues for either (a) tightening the taxonomy in a follow-up ADR, or (b) flagging tags with low inter-model agreement as confidence: low so downstream consumers can degrade gracefully.

Neither is a launch blocker. The 85–90% ceiling is good enough for the UI surfaces that motivated the audit (Round/Reveal/Recent driver chips, Insights triggers eyebrow). Surfaces that get built later on top of this data should know this caveat.