ADR-049: Scenario scheduler weighting by user's driver × trigger learning gaps¶

Status: Proposed Date: 2026-05-09 Context: The existing scenario scheduler (per ADR-005 — category balancing via least-played selection) ranks scenarios by category novelty, trigger novelty, freshness, and difficulty. Reactive arcs (per ADR-026) add a thematic-adjacency factor for narrative coherence. Both are user-pattern-aware in coarse ways (which categories has the user played), but neither weights by the model's uncertainty about how a user will decide in specific contexts.

On the May 9 cici×coachj 1:1, Cici proposed an extension to scenario selection:

"If it doesn't know enough about how the user would act in a life-or-death situation, just for example, then those games need to be pushed to the top of the queue for them. It doesn't even need to be a case of explaining that to the user — it's more just like a tool to help the agent learn more relevant things."

This is a real product idea distinct from the user-facing recommendation engine (ADR-045 v2). That engine tells the user where to play next in plain language. This ADR proposes a backend weighting — the scheduler quietly biases scenario selection toward (driver × trigger) pairs where the model has weak signal, without surfacing this as a reason to the user.

At a glance

What it decides: add a fifth, system-side scheduler factor that quietly biases scenario selection toward triggers where the model has weak signal on how this user decides — improving prediction accuracy without any user-facing copy. Proposed, not yet built.

Core mechanism — a per-user (driver × trigger) gap matrix. Cells with low evidence or low model certainty raise a trigger's selection weight (default 0.15, soft, doesn't override the existing four factors).
Trigger-side only. Driver choice is unknowable until the user plays, so only the trigger dimension can be weighted at selection time.
Stays system-side on purpose. Surfacing it would duplicate ADR-045 v2's recommendation surface; the payoff shows up as rising Accuracy, which users already see.
Rejected — replacing info-gain scoring. Learning-gap coexists with it; different concerns (user-evidence relevance vs decision relevance).
Cost/caveat — needs minimum games before activating (else the whole matrix reads as "gap" and adds noise); thresholds and decay are v1-arbitrary, tuned later.
Solo only for v1. Multiplayer scheduling conflicts with team coordination.

Decision¶

Add a fifth weighting factor to the scheduler — user-evidence learning-gap score — that biases scenario selection toward triggers where the model has insufficient signal on how this specific user decides. The factor is a soft weight, not an override; the existing four factors continue to dominate.

What gets measured¶

For each user, build a sparse 8 × 10 matrix of (driver × trigger) cells. Each cell holds: - evidence_count — number of decisions in that cell - model_certainty — derived from prediction agreement on that cell's decisions (rule + Claude both predicted correctly = high certainty; both wrong or split = low certainty)

A cell is "learning-gap-eligible" if evidence_count < 3 OR model_certainty < some threshold. The user's trigger-level gap score sums over the cells with that trigger:

trigger_gap_score(t) = sum over drivers d of (gap_weight if cell (d, t) is learning-gap-eligible)

A scenario carrying trigger t inherits a learning-gap weight proportional to trigger_gap_score(t). The driver dimension can't be weighted at selection time because we don't know which driver the user will pick until they pick — but the trigger dimension can be weighted because A.2 (per ADR-043) populated scenarios.trigger for all 895 scenarios.

How it integrates with the existing scheduler¶

Re-weight the existing factors and add the new one:

Factor	Static arcs (current)	Static arcs (proposed)
Category novelty	0.30	0.25
Trigger novelty	0.25	0.21
Freshness	0.25	0.21
Difficulty	0.20	0.18
Learning-gap	—	0.15

Reactive arcs already have a fifth factor (thematic adjacency at 0.15); learning-gap fits in alongside it as a sixth, with each component down-weighted ~20%.

Default weight 0.15 is a starting estimate. Tuning happens after a few weeks of data — same approach as the dimension distribution watch in ADR-045 v2.

flowchart LR
    A["User decisions<br/>(driver × trigger cells)"] --> B{"Cell weak?<br/>evidence < 3 OR<br/>certainty low"}
    B -- "yes" --> C["Trigger gap score ↑"]
    B -- "no" --> D["No gap weight"]
    C --> E["Learning-gap factor<br/>(0.15)"]
    E --> F["Blend with 4 existing<br/>factors → rank scenarios"]
    D --> F
    F --> G["Surface next round"]

Weak-signal triggers gain a soft selection weight; it blends with the existing factors rather than overriding them.

What stays system-side, what surfaces¶

System-side (per Cici's framing): - The scheduler quietly biases toward weak-signal triggers - No user-facing copy explains this is happening - The user just sees "a quick round" and plays it

Optional later — surfacing the reason: - If user testing shows that explaining the bias increases play rate ("we picked this one because your twin hasn't seen you under crisis yet"), a small affordance below the round can surface it. Defer this until evidence supports it; resist the urge to over-explain.

What it doesn't do¶

Doesn't replace info-gain scoring. The existing info-gain logic (decision-relevant scenarios first) still applies. Learning-gap is a user-specific extension on top.
Doesn't dominate trigger novelty. The existing trigger-novelty factor (0.21 in the proposed weighting) still rewards new triggers the user hasn't seen. Learning-gap rewards triggers where the model is uncertain, which is a different signal — a user could have played a trigger 5 times and still have weak model certainty (e.g., they keep picking different drivers). Both factors coexist.
Doesn't apply to multiplayer scenario selection. Multiplayer scheduling is constrained by team coordination concerns; per-player learning gaps would conflict. Solo only for v1.
Doesn't gate scenario availability. Even cells where the model has full certainty still surface scenarios occasionally — variety matters for engagement.

Rationale¶

Cici's intuition aligns with how good probabilistic models train: query the points of highest informational value first. A scheduler that's blind to model uncertainty wastes user playtime on scenarios that won't update the model meaningfully. Adding the learning-gap factor closes that gap without adding user-facing complexity.

The decision to keep this system-side (not user-facing) reflects two things: 1. The user-facing recommendation engine (ADR-045 v2) is already action-mapped — Accuracy → play more, Coverage → play in {weak category}, Readability → multiplayer. Adding a fourth user-facing recommendation ("play scenarios where the model is uncertain about you") clutters the surface and competes for attention. 2. The value of this factor is the model's improved prediction accuracy over time — that shows up in Accuracy going up (a dimension users do see). Users get the benefit without needing to understand the mechanism. Same way Strava doesn't tell you "we showed you a hill workout because your VO2 estimate has high variance" — it just makes the recommendations smarter.

Three implementation notes worth flagging:

The driver-side gap can't be directly addressed at selection time. Because the user's driver choice on a scenario is unknowable until they play it. We can only bias on triggers, knowing that a scenario with a learning-gap trigger gives the model some opportunity to fill in driver evidence under that trigger. Indirect, but real.

The cell evidence threshold (evidence_count < 3) is arbitrary for v1. Worth tuning when distributions stabilize. If users have 80 cells with evidence_count = 0 (because they've only played 5 scenarios), the entire matrix is "gap" and the factor adds noise. Probably want to require minimum total games played (e.g., 20) before the factor activates.

Decay matters. A cell that had high model certainty 3 months ago might not now (the user's behavior may have shifted). Either decay evidence_count and model_certainty over time, or weight recent decisions more heavily. v1 can use a simple "last 50 decisions only" window; revisit if the data suggests it matters.

Alternatives Considered¶

Surface the bias to the user as a separate recommendation. Rejected — duplicates ADR-045 v2's surface and competes for user attention. The whole point of system-side is that the user gets the benefit without the complexity.
Replace info-gain scoring entirely with learning-gap weighting. Rejected. Info-gain is decision-relevance ("is this scenario good to ask?"). Learning-gap is user-evidence-relevance ("does this scenario fill a gap for this specific user?"). Different concerns; they coexist.
Use rule engine confidence directly as the cell certainty proxy. Considered. The rule engine already produces a per-prediction confidence number; 1 - confidence × correctness gives a per-decision uncertainty. Aggregating across decisions in a cell gives model_certainty. This is the implementation path for v1.
Surface this as a system-tuning lever in the admin dashboard. Worth doing eventually so we can A/B the weight value (0.15 vs 0.20 vs 0.10). Not blocking for v1.
Amend ADR-005 instead of writing a new ADR. Considered. Decided no — the new behavior is meaningful enough that future readers will find it more easily as its own ADR. ADR-005 stays the canonical "category balancing" decision; this one is the canonical "user-specific evidence-gap weighting" decision.

Discussion¶

The framing distinction this ADR makes — system-side scheduling intelligence vs user-facing recommendation copy — is worth preserving as a general principle for future product decisions. Sync has multiple surfaces where the same data could either drive smarter machine behavior or be surfaced as user-readable advice. Defaulting to "smarter machine" is the right call when: - The benefit shows up downstream as a metric users do see (here: Accuracy goes up) - Adding the surface to the UI would compete with existing user-facing recommendations - Explaining the mechanism doesn't itself increase engagement

That's a useful frame to keep handy — most future scheduler tweaks will fit this shape, not the user-facing-rec shape.

The sync_score-as-trust-threshold framing from the same May 9 conversation (see project_sync_score_as_trust_threshold memory) is the opposite call — there, surfacing the threshold to the user is exactly the point, because the user benefit (knowing what their agent can do without asking) requires explaining. Different surfaces, different framings.

This ADR doesn't address the cell decay question (recent decisions weighted more than old ones) — that's a tuning detail that needs production data before it can be answered well. v1 can use a simple last-N-decisions window and revisit.

Consequences¶

One new helper module at src/lib/scheduler/learning-gap.ts (proposed): computes the user's (driver × trigger) cell matrix from choices joined to scenarios + results.ai_*_prediction + ai_*_confidence. Returns trigger-level gap scores ready to inject as a scheduler factor.
No new tables, no migrations. All input data already exists post-ADR-043 backfill.
Existing scheduler at src/lib/game/campaign-scenario-selector.ts (and equivalents) gets the new factor wired in. Re-weight the existing factors per the table above.
Solo only for v1. Multiplayer scenario selection is out of scope.
Watch metric: Accuracy (per ADR-045 v2) growth rate vs control. If the learning-gap weighting works, users with the new scheduler should see Accuracy improve faster than users without it (could A/B-test for confirmation, but at current scale ~50 users a single-arm rollout is fine).
Tuning lever: LEARNING_GAP_WEIGHT constant (default 0.15). Adjustable without code changes if surfaced in the admin dashboard later.

Key files (proposed): - src/lib/scheduler/learning-gap.ts — gap-score helper - src/lib/game/campaign-scenario-selector.ts — wire the new factor - src/lib/ai/scenario-generator.ts — no change; A.2 trigger column is already populated - src/app/admin/agent-eval/page.tsx — could grow a "scheduler diagnostics" panel showing the gap matrix per user (later, optional)