Skip to content

ADR-038: Upgrade primary AI model from Sonnet 4 → Sonnet 4.6

Status: Accepted Date: 2026-05-04 Context: The production prediction, learning-note, classification, campaign-advancement, and arc-generation paths were all pinned to claude-sonnet-4-20250514 (Sonnet 4, May 2025) — roughly a year behind the current Sonnet frontier. Anthropic has since shipped Sonnet 4.6 (claude-sonnet-4-6) and Opus 4.7 (claude-opus-4-7). The shadow-eval harness was on the same Sonnet 4 model. Time to refresh.

Decision

Upgrade the primary model in src/lib/ai/claude-client.ts and the eval model in src/lib/agent/eval.ts from claude-sonnet-4-20250514 → claude-sonnet-4-6. Leave summary on claude-haiku-4-5-20251001 (already current). Scripts in scripts/ are out of scope for this ADR — they're one-off or pinned for reproducibility of historical passes.

Rationale

  • Sonnet 4 is ~1 year stale. Free quality lift available by moving to the current Sonnet generation at unchanged cost-class.
  • Same price tier. Sonnet 4.6 is priced identically to Sonnet 4 ($3 input / $15 output per MTok), so no cost-model rework required for the reactive arc cost ceiling tracked in the monetization roadmap.
  • Single source of truth. All prediction and arc-generation routes import MODEL_VERSION from claude-client.ts, so one edit propagates across solo-predict, multiplayer-predict, learning-note, classify-driver, campaign advance, generate-arc, and speed-round paths.
  • Eval moves with live. The shadow-eval harness exists to measure agreement-rate between live predictions and the eval conditions (A/B/C). If live and eval drift to different models, agreement-rate stops being a coherent signal — so they're upgraded together. The trade-off is that pre-2026-05-04 agreement-rate trends are no longer comparable to post-upgrade trends; baselines reset from this date.

Alternatives Considered

  • Upgrade everything to Opus 4.7. Highest quality but 5× the cost (Opus is $15/$75 per MTok input/output). For per-game costs that translates to roughly +$0.88 per solo game and several dollars per reactive campaign. Rejected as default; held in reserve as a future option for the prediction path specifically if Sonnet 4.6 accuracy plateaus.
  • Mixed strategy — Sonnet 4.6 for stories/scenarios, Opus 4.7 for predictions and learning notes. Considered explicitly in the deliberation. Rejected for now because: (a) the two-stage prediction system already arbitrates Claude against the rule-based predictor, dampening how much pure LLM quality matters at the margin; (b) Sonnet 4 → 4.6 is itself a substantial jump and likely captures most of the realizable gain; (c) easier to defer the Opus question until Sonnet 4.6 baseline accuracy is in the eval dashboard.
  • Stay on Sonnet 4. Rejected — staying a year behind the frontier with no upside.
  • Pin scripts to Sonnet 4.6 too. Deferred. Several scripts (e.g., scripts/audit-static-arc.ts, scripts/generate-chapter-synopses.ts) were used for specific historical passes (e.g., the Apr 23 static arc rewrite). Bumping their model retroactively would make those passes non-reproducible. Future script invocations can be updated case-by-case.

Discussion

The core tension was cost vs quality on the prediction path specifically. Story/scenario generation has a clear ceiling — Sonnet is fine because the output is judged by the writer, not by accuracy metrics. Predictions are different: better calibration there directly improves the Sync Score's signal quality, which is the whole product.

The user initially leaned toward Opus 4.7 for predictions, asking how much more expensive Opus calls would be. Once the 5× multiplier was made concrete (~$0.10/call vs ~$0.02/call, ~$0.88 added per solo game, multiplied across multiplayer and campaigns), the calculus shifted. The deciding argument was that the two-stage arbitration system already softens the impact of LLM choice on the final prediction — so Opus's marginal accuracy gain doesn't propagate 1:1 to final Sync Score quality. Better to capture the Sonnet 4 → 4.6 jump first, observe eval impact, and revisit Opus from a position of real data rather than estimates.

The eval-model question was settled quickly: agreement-rate trends are only meaningful when live and eval share a model, so they move together. The trade-off (resetting trend baselines) was accepted as the cost of doing business.

Consequences

  • All AI prediction, learning-note, classification, campaign, and speed-round paths now run on Sonnet 4.6.
  • Shadow-eval harness runs on Sonnet 4.6; pre-2026-05-04 agreement-rate baselines should not be compared against post-upgrade numbers.
  • Cost profile unchanged at the per-token level; absolute spend may shift slightly if Sonnet 4.6 produces different output-token counts on average.
  • Watch the agent-eval dashboard over the next 2–4 weeks for accuracy and agreement-rate movement. If accuracy noticeably improves, this validates further upgrade caution being unnecessary; if it plateaus or regresses on certain scenario types, that's the trigger to revisit Opus 4.7 selectively for predictions.
  • MODEL_VERSION is committed into prediction commitment payloads — old commitments will retain the Sonnet 4 string, which is correct historical record.

Key files: - src/lib/ai/claude-client.ts — primary MODEL_VERSION constant; consumed by all prediction routes - src/lib/agent/eval.ts — EVAL_MODEL / EVAL_MODEL_VERSION for the shadow-eval harness - src/lib/agent/summary.ts — unchanged (already on Haiku 4.5)