Skip to content

ADR-001: Two-Stage Prediction System

Status: Accepted Date: 2026-02-10 Context: The AI prediction system needs to balance accuracy, cost, and interpretability. Pure Claude predictions are expensive and opaque. Pure rule-based predictions lack nuance for complex scenarios.

Decision

Predictions use a two-stage pipeline:

  1. Stage 1 — Rule-based (free): If the player has 3+ games, predictFromProfile() scores each option against the player's driver distributions, context rules, and keyword preferences. Returns a prediction with confidence score.

  2. Stage 2 — Claude validation (costs tokens): An enhanced prompt with full profile context and the rule prediction is sent to Claude. Claude either agrees (confidence boosted +0.10–0.15) or disagrees (returns its own higher-confidence prediction).

Multiplayer optimization: if the rule engine hits >=0.5 confidence, Claude is skipped entirely (cost savings for batch predictions across multiple players).

Each prediction records its method ('rule-based' | 'claude' | 'two-stage') for accuracy analysis.

Rationale

Simulation testing showed rule-based predictions outperform Claude-with-notes in disagreement cases — the rule engine won 70.2% of the time vs Claude's 29.8%. This means the rule engine should lead, with Claude as a validator rather than the primary predictor.

The two-stage approach also means the system degrades gracefully: if the Claude API is down or rate-limited, rule-based predictions still work.

Alternatives Considered

  • Claude-only: Higher cost per prediction (~$0.01–0.02), no local fallback, and empirically less accurate than the rule engine for players with sufficient history.
  • Rule-only: Misses nuance in novel scenarios where keyword matching fails. Claude adds value for edge cases and when player behavior is shifting.
  • Ensemble averaging: Averaging rule and Claude predictions was considered but rejected because the rule engine's discrete option scoring doesn't produce probability distributions compatible with Claude's natural language reasoning.

Consequences

  • Prediction method is tracked per game, enabling ongoing analysis of which stage drives accuracy.
  • The rule engine must be maintained alongside the Claude prompt — changes to driver keywords or scoring weights affect both stages.
  • Multiplayer's confidence threshold (0.5) for skipping Claude is a tunable parameter that trades accuracy for cost.

How We Got Here

The system started as Claude-only predictions. Every game sent the player's full history to Claude and asked it to predict. This worked but was expensive and had no fallback.

The v2 player model (driver distributions, context rules, keyword preferences) was built to enable rule-based predictions. Once the rule engine existed, the question was: does it replace Claude or complement it?

Simulation testing answered this. Multiple persona types were run through hundreds of simulated games, comparing rule-based vs. Claude predictions. The key finding: when the rule engine and Claude disagreed, the rule engine was right 70.2% of the time. Claude's strength was narrative reasoning, but for predicting specific option choices, behavioral patterns outperformed narrative interpretation.

This was surprising — the expectation was that Claude's ability to "understand" the scenario would give it an edge. Instead, the structured data (driver distributions, keyword preferences, context-triggered behavior shifts) turned out to be more predictive than natural language reasoning.

The two-stage design followed from this: let the rule engine lead (it's free and more accurate), then use Claude as a validator that can catch edge cases where keyword matching fails. The multiplayer optimization (skip Claude if rule confidence ≥ 0.5) was added because batch-predicting all players in a multiplayer session made Claude costs add up quickly.

A critical discovery in Week 10 changed the context: the rule engine had been broken in production since launch. An RLS bug meant player_decision_profiles updates were silently failing — games_analyzed stayed at 0 for every user. The two-stage system was running as Claude-only with no behavioral data accumulating. All simulation results reflected ideal conditions, not real-world performance. The fix shipped, but we're still accumulating real data on how the two-stage system performs with functioning profiles.

Key files: - src/lib/ai/player-model.ts — predictFromProfile() (rule engine) - src/app/api/ai/solo-predict/route.ts — two-stage orchestration - src/app/api/ai/multiplayer-predict/route.ts — batch prediction with confidence-based Claude skip