ADR-057: Integrate post-reveal reaction and note into the prediction pipeline¶

Status: Accepted (Phase A shipped 2026-05-15; Phase B remains deferred) Date: 2026-05-15 Context: ADR-024 added the post-reveal quick-tap reaction (gets_me / off_track / interesting) and a free-text note field as Signal 2 of the 12-signal judgment layer. The collection side shipped: reactions and notes save to results.post_reveal_reaction and results.post_reveal_note, and aggregate counters update on player_decision_profiles.self_awareness.resonance_stats. The integration side was deliberately deferred under the "collect now, separate signal from noise later" principle — with limited gameplay volume, we couldn't yet tell which signals carried predictive weight, so we paid the cheap collection cost and left the integration question for later. On 2026-05-15 a concrete miss surfaced. On a values-culture scenario ("The Children's Question"), the twin predicted "Make an exception" at 38% confidence with an explicit hedge ("Interestingly, your choice history suggested you might pick 'Enforce the rule' instead"). The player chose "Split the family temporarily" and tagged the reasoning "Interesting" with the note: "I act very differently in resistance/underground network scenarios than in boardroom situations." This is a high-bandwidth domain-conditional self-report — the exact shape of signal the deferral was supposed to grow into. Today it sits unused. The next time the same player hits a boardroom scenario, the twin will average underground-network and boardroom behavior back into a single flat profile.

At a glance

What it decides: The post-reveal reaction (gets_me / off_track / interesting) and free-text note from a player's prior games now feed the learning-note prompt, so the twin can read the player's own theory about their behavior — the highest-bandwidth signal Sync collects. Profile restructuring (Phase B) stays deferred.

Phase A shipped: route the reaction through the learning note, not the prediction prompt directly — the learning note produces a durable prediction_rule, so a correctly-generalized reaction shapes all future predictions instead of a single one-off.
Phase B deferred and eval-gated — conditioning the profile on driver/trigger axes (not scenario category — boardroom/resistance/sci-fi don't transfer to real-life agent use; drivers do). Ships only if Phase A's eval shows reaction-text alone can't close the domain-conditioning gap.
Eval result: 4 of 5 cases visibly sharper — statistical A/B wasn't viable (only 5 notes, 0 off_track), so a pre-committed qualitative case study substituted. Caveat: richer context introduced one stray over-applied modifier — over-application risk to watch.
Humility clause — Sync measures driver patterns under forced-tradeoff framings; it does not claim to know real high-stakes conduct the data can't reach. The note is one signal, not authoritative (watch: players gaming the note field). No schema changes, no new API calls.

flowchart LR
    R["Prior games:<br/>post_reveal_reaction + post_reveal_note (cap 20)"] --> LN["Learning-note prompt"]
    C["Recent choices + driver/trigger + profile"] --> LN
    LN --> PR["prediction_rule (durable)"]
    PR --> NEXT["Next-game prediction"]
    R -.->|"Phase B (deferred, eval-gated)"| PB["Driver/trigger-conditioned profile buckets"]
    PB -.-> NEXT

Phase A routes prior reactions through the learning note so they shape the durable prediction rule; Phase B (dashed) would condition the profile on driver/trigger axes and remains deferred behind the eval gate.

Decision¶

Wire post_reveal_reaction and post_reveal_note from prior games into the learning-note generation prompt as Phase A. Defer per-category profile restructuring (Phase B) until the eval harness can compare prediction accuracy with vs. without the reaction signal included.

Phase A — Learning-note integration (immediate)

Extend LearningNoteContext in src/lib/ai/learning-notes.ts with a priorReactions field: an array of { scenarioTitle, category, reaction, note } for the player's recent N games where reaction is non-null. Cap N at 20 to keep the prompt bounded.
The query in src/app/api/ai/learning-note/route.ts selects post_reveal_reaction and post_reveal_note alongside the existing learning-note fields.
The prompt instructs Claude to weight off_track reactions as direct refutations of past predictions, interesting as identifying a signal the player wants surfaced, and free-text notes as the player's own theory about their behavior — to be tested against the choice pattern, not accepted at face value.

Phase B — Driver/trigger-conditioned profile buckets (deferred, eval-gated)

The current player_decision_profiles.category_patterns is a single EMA-weighted distribution averaged across all scenarios. The right axis for conditional profiles is not the narrative-setting category (boardroom, resistance, sci-fi, family) — those don't transfer to real-life agent use where decisions are about health, finance, or scheduling, none of which match Sync's settings. The right axis is the abstract decision-structure layer: ADR-043's driver and trigger tags, which are scenario-agnostic and transfer across settings.

Concretely, Phase B would condition the behavioral model on driver and/or trigger dimensions rather than scenario category — e.g., "Jonathan's choice distribution when authority-threat is triggered" or "Jonathan's pattern on collective-welfare-vs-individual-autonomy tradeoffs," not "Jonathan in boardroom scenarios." This is the cut that survives the bridge to real-life agent decisions, which won't carry Sync's narrative wrappers but will carry the same underlying decision structures.

This is an architectural change to the profile schema and to updatePlayerProfile / predictFromProfile / formatProfileForPrompt. Defer until the Phase A eval shows either that (a) reaction-text alone closes the domain-conditioning gap (Phase B unnecessary) or (b) it doesn't (Phase B justified).

Humility clause — what Sync claims, what it doesn't

The integration in this ADR — and Sync more broadly — measures driver patterns under counterfactual forced-tradeoff framings. It does not claim to know what a player would actually do under real high-stakes conditions they've never been in. A player who chooses "radical action" in a resistance-network scenario has revealed a driver weighting (e.g., collective-welfare dominant under crisis framing), not a behavioral prediction about their actual conduct in an actual hacktivist cell. The agent uses driver patterns to inform recommendations on the decisions in front of the user; it doesn't tell the user who they "really are" under conditions the data can't reach. This humility is what keeps Sync's value claim defensible: we measure behavior-under-forced-tradeoffs, not character.

Eval gate

The ADR-024 §9 A/B/C harness is already live in src/lib/agent/eval.ts, with evaluation_snapshots freezing profile state before live updates (per ADR-044's snapshot fix). For Phase A's eval:

Run learning-note generation on a held-out set with and without priorReactions included in the prompt.
Because learning-note generation runs at temperature 1.0, single-pass comparison is dominated by Claude variance — run each condition 5× per game and average.
Compare the resulting prediction rules' accuracy against the player's actual choice on the next game.
Stratify by category and by driver/trigger — overall accuracy, plus the values-culture and authority-threat-triggered subsets where the May 15 miss occurred.
Report effect size with a confidence interval. Honest reporting: at current data volume (~200–400 games across active accounts), the result may be inconclusive — itself a finding that informs whether to escalate to Phase B.

If accuracy is unchanged or worse, do not merge — revisit prompt shape or escalate to Phase B.

Eval result (2026-05-15): Statistical A/B was not viable — only 5 results in the database had non-empty post_reveal_note (and 0 had off_track), so any percentage-based "accuracy lift" claim would be dominated by Claude variance at temperature 1.0. Substituted a qualitative case-study: for each of the 5 note-bearing games, generated the learning note with and without the new reactions section, side-by-side. Pre-committed decision rule: ship if reactions-aware version names a domain/stakes/frame conditional pattern that baseline misses on 3+ of 5 cases.

Result: 4 of 5 visibly sharper. The wins were traceable to specific reaction/note text the baseline prompt structurally cannot see — systems-change as a driver beyond category (Robin Hood Question), cross-game integration of prior feedback (Shift Allocation Crisis), occasion-noise / instability disclosure (Safety Training Crisis), and explicit domain-conditioning (Children's Question — the trigger case). The 5th case (The Old Guard) had a confirmation-only note ("on point, the risk aversion is real") and correctly produced a confirmation-only refinement — control behaved as expected.

Two honest caveats from the case-study: (1) the real baseline-Claude is more sophisticated than initially rendered inline; it extracts a lot of meta-pattern from rationale text alone. The gap between baseline and new is smaller than first impressions suggested, but its direction is consistent. (2) The richer context introduces over-application risk — Case 3's new version manufactured a stray systems_change_opportunity modifier that didn't fit the scenario, evidently cross-applying Case 1's pattern. Worth watching in production but not a blocker. Phase A shipped at 2026-05-15 with this eval result on record.

Rationale¶

The note text is the highest-bandwidth signal we collect. Choice + driver + trigger is structured but lossy. The free-text reaction note is the player explicitly naming a domain-conditional pattern in their own behavior — the exact signal a flat profile cannot represent. Discarding it while spending Claude tokens on EMA-summarized choice patterns is the wrong cost trade.

Cheap before expensive. Phase A is a prompt change plus a query column addition. Schema is unchanged (columns already exist). Phase B touches the profile model, the predictor, the prompt builder, the migration path for existing profiles, and the rule-engine path. Phase A is hours; Phase B is days and carries migration risk.

Honors ADR-024's "collect, then validate." The deferral logic was that we don't yet know which signals predict well. The eval gate respects that — Phase A ships only if it demonstrably improves predictions, and Phase B is gated on Phase A's results. We don't commit to architecture before the data justifies it.

The May 15 miss is the canonical case the system was built to catch. The 38% confidence with the hedge ("history suggested you might pick 'Enforce the rule'") means the rule-engine signal was correct and the LLM overrode it with a frame heuristic. That's a recoverable miss if the next prediction can read "I act differently in resistance scenarios" as a domain-conditional pattern. Without integration, the same miss is structurally guaranteed to repeat.

Prediction-time vs. learning-time integration. Integrating into the learning-note prompt (not the prediction prompt directly) is deliberate. The learning note already produces a structured prediction_rule that feeds future predictions. Routing the reaction through the learning note means it shapes the rule, which is durable, rather than shaping a single prediction in isolation. If the learning note generalizes the reaction correctly, all future predictions benefit; if it fails to generalize, the failure is localized to one row and visible in the rule text for debugging.

Alternatives Considered¶

Do nothing (status quo ADR-024 deferral): Rejected. The deferral made sense when no integration case had surfaced. May 15 is the surfacing case — the signal was high-bandwidth, the miss was structural, and the cost of one more deferral cycle is more repeated misses on domain-conditional behavior.
Skip Phase A; go straight to per-category profile buckets: Rejected. We don't yet know if the reaction-text alone is sufficient. Architectural change first, then prompt change, is the wrong order — it commits the schema before validating the signal.
Include reaction in the prediction prompt directly, bypassing the learning note: Rejected for the first pass. The learning note is the durable abstraction — the prediction prompt sees recent choices + profile + the learning notes' accumulated rules. Routing reactions through the learning note lets them shape the durable layer instead of being a one-shot input. If Phase A succeeds and we still see misses, adding to the prediction prompt is a small follow-on.
Build a separate reaction-conditioned predictor (third ensemble member alongside rule + Claude): Rejected. We already have ensemble complexity (ADR-044); adding a third predictor before validating that reaction text adds signal at all is premature.
Wait for more games before integrating anything: Rejected. We're at the volume where individual misses are diagnostic. Waiting for a statistically significant aggregate gap means waiting for many more identical misses.

Discussion¶

The deferred-integration discipline from ADR-024 is real and should not be casually broken. The argument for breaking it now is not "we have lots of data" — we still don't. The argument is that we have a specific, diagnostic miss where the signal we're sitting on would plausibly have changed the prediction. That's a different evidentiary bar than "aggregate stats show feedback helps."

The eval gate is doing the work of preserving the discipline. Phase A doesn't ship blind; it ships only if the harness confirms the diagnostic intuition. If the eval is flat, we learn that free-text reactions don't generalize well from one player to a refined rule — and that's itself a valuable finding that informs Phase B's design (maybe per-category buckets are the only way to capture domain-conditioning, because text generalization fails).

The domain-conditioning frame is what makes this ADR distinct from "integrate self-awareness signals generally." The player's note was not "the AI got my reasoning wrong" (which off_track would have captured structurally). It was "the AI is missing that my behavior is domain-conditional." That's not a self-awareness signal alone — it's a structural claim about how to read the profile. Phase B exists in this ADR because the player's note pointed at a structural gap, not just a per-game miss.

One open question deferred to the eval: does interesting differ meaningfully from gets_me as a signal? Both indicate the player isn't refuting the reasoning. The note text may be the discriminator — interesting + note is "the AI was close enough to provoke a refinement," while interesting alone may be near-noise. If the eval shows note-bearing interesting is high-signal but note-less interesting is not, the prompt should weight them differently.

Phase B's conditional-profile design is intentionally left at the axis-choice level (driver/trigger, not scenario category) and deferred on the exact mechanism. The choice between (a) per-driver EMA buckets, (b) a driver-conditional multiplier on the existing flat profile, or (c) a learned conditional adjustment from history is itself a design decision that should be made after Phase A's eval clarifies how much of the gap reaction-text alone can close.

The setting-variety question. Sync's scenarios span boardroom, resistance/underground, sci-fi command, family, and progressively more fantastical settings (vampire theater, Roman history) — originally added for engagement. The question of whether this variety helps or distracts the signal model resolves cleanly once Phase B is framed along driver/trigger axes: setting variety becomes signal-positive, because a driver pattern surviving translation across radically different narrative wrappers is much stronger evidence of a stable driver than the same pattern observed only in one setting. Setting variety stress-tests driver stability. The real corpus bias to watch is not setting-variety but stakes-uniformity — Sync's scenarios skew heavily to crisis-mode/life-or-death framings, leaving the mundane-decision regime (where most real agent use lives) under-represented. This is a separable problem from ADR-057; tracked as a future direction below.

Future direction (separate ADR): Structural-stakes tagging. Tag scenarios on what is structurally at risk (lives, livelihoods, relationships, identity, money) and magnitude, as a corpus property. This enables stakes-conditional pattern recognition ("rule-preservation increases as structural stakes rise") and surfaces coverage gaps ("profile has 47 high-stakes-life games, 3 mundane-decision games — recommendations on routine decisions are extrapolating from thin data"). Subjectivity is contained by tagging the scenarios, not the players' felt experience. The bridge to mundane-stakes real-life decisions is ADR-039's decision-capture primitive — capture decisions made through the agent skill, not just through games. Games train the driver model; agent-skill decisions become ground truth at the stakes levels games can't reach.

Consequences¶

LearningNoteContext gains priorReactions — typed array of recent reaction events; the learning-note prompt is extended to instruct Claude how to weight each reaction class.
/api/ai/learning-note route fetches reaction columns — the existing results query is extended to select post_reveal_reaction and post_reveal_note for the player's recent N games.
Eval harness must run before merge — concrete pass/fail criterion on values-culture and resistance/network subsets, plus overall accuracy.
Phase B is explicitly deferred — category_patterns remains a single flat distribution until Phase A's eval result is in.
No schema changes — post_reveal_reaction and post_reveal_note already exist from ADR-024's migration 067.
No new API calls — the learning-note Claude call already runs once per game; adding reaction text to its prompt is a token-count change, not a call-count change.
Watch for prompt bloat — capping priorReactions at 20 keeps the addition bounded; revisit if note text is long enough to push the learning-note prompt past comfortable token limits.
Watch for the player learning to game the note field — if reactions start shaping predictions, the note becomes a manipulable surface. ADR-024's bias philosophy applies: the note is one signal among many, not authoritative.
Open follow-on: if Phase A succeeds, decide whether to also include reactions in the prediction prompt directly (not just via learning notes) — likely yes for off_track reactions on the immediately preceding game, where freshness matters most.
Phase B reframe (this ADR, 2026-05-15 revision): conditional profiles cut along driver/trigger axes (ADR-043) rather than scenario category. Setting categories don't transfer to real-life agent use; drivers do. This changes the Phase B scope from a schema split on category to a schema split on the abstract decision-structure dimensions.
Stakes-tagging queue item: structural-stakes tagging is the next adjacent improvement after Phase A/B settles. Tracked as a future ADR, not part of 057.

Key files: - src/lib/ai/learning-notes.ts — LearningNoteContext type extended; buildLearningNotePrompt instructed on reaction weighting - src/app/api/ai/learning-note/route.ts — recent-games query selects reaction + note columns; assembles priorReactions - src/lib/ai/prompt-builder.ts — no change in Phase A; revisit in Phase A follow-on if needed - src/lib/ai/player-model.ts — no change in Phase A; Phase B would touch updatePlayerProfile / predictFromProfile / formatProfileForPrompt - (eval) the dual-predictor harness from ADR-044 — extended with the Phase A comparison