ADR-006: Confidence Capping for New Players

Status: Accepted Date: 2026-02-10 Context: The AI prediction system can produce high-confidence predictions even for brand new players if a scenario happens to strongly match keyword patterns. This is misleading — a DTA claiming 80% confidence after 1 game erodes trust.

Decision

Prediction confidence is capped based on games analyzed:

• < 3 games: Rule engine returns null (no prediction attempted). Claude-only predictions capped at 0.40.
• 3–5 games: Max confidence 0.50.
• 6+ games: Normal confidence calculation (capped at 0.85 base, then multiplied by model_confidence which maxes at 0.95).

The rule engine (predictFromProfile()) returns null for < 3 games. The model confidence formula itself also scales with experience: the game component contributes min(0.3, games × 0.02), meaning 15 games are needed to max out the experience factor.

Rationale

Trust is earned. A DTA that says "I'm 80% confident I know what you'll choose" after 2 games would feel presumptuous and wrong. If it happens to be right, the player thinks "lucky guess." If wrong, the player loses trust in the system. Low confidence that gradually increases mirrors the natural process of getting to know someone — the DTA should signal its own uncertainty honestly.

Alternatives Considered

• No capping (let confidence reflect the math): Rejected because keyword matching can produce spuriously high scores on thin data.
• Hide confidence entirely for new players: Considered but the confidence display is part of the learning experience — seeing it grow from 30% to 70% is motivating.
• Linear scaling (confidence × games/10): Too aggressive — a player at game 5 would have their confidence halved even if the prediction is strong.

Consequences

• New players see low-confidence predictions that feel honest rather than overconfident.
• The sync score delta is confidence-weighted, so early wrong predictions have small negative impact.
• The 3-game minimum for rule-based predictions means the first 2 games are Claude-only (higher token cost per prediction during onboarding).

How We Got Here

Early playtesting revealed a trust problem: the AI would sometimes predict a new player's choice correctly on game 1 or 2 and display "85% confidence." The player's reaction wasn't "wow, the AI knows me" — it was "that's creepy" or "that's obviously a lucky guess." When the AI got it wrong at high confidence (which happened equally often), the reaction was worse: "this system doesn't work."

The insight was that first impressions of AI confidence set the tone for the entire relationship. A DTA that starts humble ("I'm only 35% sure, but I think you'll pick B") and gradually becomes more confident ("After 20 games, I'm 72% sure you'll pick A in governance scenarios") mirrors how trust actually develops between humans. You don't trust a new colleague's judgment on day one — you watch them, test them, and trust builds through demonstrated competence.

The specific thresholds (<3 games: null/0.40, 3-5 games: 0.50, 6+: normal) were chosen pragmatically. Three games is the minimum to establish any pattern — with 4 options per scenario, 2 games could be coincidence. Five games provides enough data for the rule engine to have meaningful keyword and driver distributions. After 6 games, the model confidence formula's built-in scaling (game experience component, category coverage) provides natural restraint.

The confidence cap also solved a scoring problem: sync score changes are confidence-weighted. Without capping, an overconfident wrong prediction on game 2 could tank a player's score before the DTA had any real understanding, creating a negative first impression of the sync score itself.

Key files: - src/lib/ai/player-model.ts — predictFromProfile() returns null for < 3 games; calculateModelConfidence() scales with experience - src/app/api/ai/solo-predict/route.ts — confidence cap application