ADR-044: Always Run Both Predictors (Rule + Claude) — Remove Rule-Confidence Short-Circuit

Status: Accepted Date: 2026-05-06 Context: The multiplayer prediction loop ran a two-stage system: STAGE 1 generated a rule-based prediction from player_decision_profiles (free, no API call); if rule confidence was ≥0.5, the rule prediction was used directly and Claude was never called. STAGE 2 (Claude) only fired when the rule wasn't confident or when the player had too few games for the rule engine to engage. This optimization saved inference cost on routine cases — but it also meant the rule-vs-Claude head-to-head comparison silently excluded every case the rule engine was opinionated about. ADR-010's deferral rested on simulation data showing rule won 70.2% of disagreements, but that finding could never be tested in production because production never produced rule-vs-Claude pairs on the rule's strong cases. A real-user-data check on 2026-05-06 found 0/53 multiplayer rows had a breakdown captured (vs. 16/16 solo rows where the breakdown was always written, because solo doesn't have the same short-circuit).

Decision

Remove the rulePrediction.confidence >= 0.5 short-circuit in multiplayer-predict/route.ts. Always call Claude when a Claude path is reachable (i.e., always, since profiles with <3 games already fall through to Claude). The user-facing ensemble logic is unchanged: Claude's prediction remains the default, with rule prediction shifting confidence on agreement and adding a user-facing note on disagreement. Both predictions are surfaced in the API response as predictionBreakdown: { rule, claude } and persisted to ai_predictions.scenario_context for cache replay and to results.ai_rule_prediction / results.ai_claude_prediction on session complete.

Rationale

The user's stated bet (see memory): in the long run, LLM prediction will outperform behavioral rule prediction. To prove or disprove this without bias, we need rule + Claude predictions on the same scenarios — including the cases the rule engine is most confident about, since those are exactly the cases where the rule's premise is most testable.

The cost trade-off is small at current scale. Multiplayer Claude calls already happen for ~77% of rounds (Stage 2). Going to 100% adds ~30% to multiplayer prediction inference cost. At ~1 multiplayer round/day in early May 2026, this is rounding error.

The data trade-off is large. The previous architecture produced an asymmetric dataset: the rule engine got the easy cases (high-confidence patterns it had already seen) and Claude got the hard ones (new players, low-confidence profiles). Any apparent rule-engine win in that data is partly a selection effect, not a method comparison. Removing the short-circuit lets both methods play every scenario.

Alternatives Considered

• Sample at 30–50% of rule-confident rounds. Lower cost, statistical validity for the comparison if N is large. Rejected because at current volume, sampling halves an already-tiny dataset; cost savings are negligible and quality matters more.
• Use the eval harness (§9 Conditions A/B/C) for the comparison. Rejected because the eval harness compares context bundles (personal_context vs. Sync profile vs. both), all using LLM. It does not run the rule engine and cannot answer the rule-vs-Claude question.
• Replay rule predictions retroactively from historical data. Rejected because player_decision_profiles is updated in-place — we don't have profile snapshots from the moment of past predictions, so any retroactive predictFromProfile call would be against a different profile than the one in effect at the time. (The evaluation_snapshots table fixes this going forward, but didn't exist for the historical rounds in question.)
• Leave as-is and document the bias. Rejected because the comparison is a load-bearing question for the value proposition of the LLM judgment layer.

Discussion

The discovery flow mattered. Initial investigation framed this as "the scoring pipeline is broken — ai_prediction_results is empty." The deeper finding was that the comparison columns on the results table existed and the solo flow populated them — but the multiplayer flow was structurally biased. The ai_prediction_results table is unused and probably should be dropped or repurposed; the results table already has the columns we need.

The user explicitly weighed cost vs. quality: "I do want quality, and yeah it would be nice not to spend extra money but quality matters too." This pushed the choice from "sample to control cost" to "always run both." Captured as a durable preference: when comparison architectures trade off cost for data quality, default to data quality at current volumes. Tunable later if cost becomes real.

A subtle point: this decision partially contradicts ADR-001's two-stage prediction system. ADR-001 framed the two-stage approach as a cost-optimization plus accuracy boost. The cost half is no longer in effect for production multiplayer rounds. The accuracy half (rule prediction informing Claude's prompt and shifting final confidence on agreement) is preserved. ADR-001 stays Accepted; this ADR refines its scope.

Consequences

• Every multiplayer round contributes one rule-vs-Claude head-to-head datapoint (when player has ≥3 games for the rule engine to engage). Solo flow already worked correctly; no change there.
• ai_predictions.scenario_context JSONB now includes predictionBreakdown for cache replay across clients (preserves the ADR-030 invariant that all clients see the same prediction).
• ~30% cost increase on multiplayer prediction inference. Monitor if multiplayer volume grows substantially.
• Pre-fix multiplayer data is unsuitable for rule-vs-Claude comparison (selection bias). Post-fix data is the comparison surface; treat 2026-05-06 as the trend baseline.
• ADR-010's underlying behavioral-vs-LLM question becomes testable for the first time. Trigger metric: rule wins ≥55% of head-to-head disagreements after ≥30 disagreement events → behavioral signal vindicated; Claude wins → LLM layer justified.

Key files: - src/app/api/ai/multiplayer-predict/route.ts — short-circuit removed; PredictionBreakdown added to response and persisted to scenario_context - src/app/api/sessions/complete/route.ts — extended aiPredictions type; writes rule/claude breakdown columns on results insert - src/app/solo/page.tsx — already correct, no change - docs/decisions/010-decision-principles-deferred.md — closed Superseded; references this ADR - docs/decisions/001-two-stage-prediction.md — scope refined by this ADR (cost-optimization half deprecated; accuracy half preserved)