Skip to content

ADR-024 classify-driver backfill — pre-promotion accounts

Status: ✅ Complete (2026-05-31). 5 accounts backfilled. Classifications produced inline (no API spend) and folded via the script's cache; see "Results." Date: 2026-05-31 Script: scripts/backfill-classify-driver.ts Relates to: ADR-024 (judgment layer signals), docs/research/foundations/measurement-schema.md (SPOF "contaminated vs missing" finding, line ~264)

The gap

ADR-024 (promoted 2026-04-12) extended the classify-driver Claude call to emit reasoning_mode, attention_focus, and a semantic driver per round. Those outputs feed three per-player aggregates on player_decision_profiles:

Aggregate column ADR-024 signal Schema primitive
reasoning_style_distribution Signal 1 (Reasoning Style) E15
attention_distribution Signal 4 (Attention Model) E16
authenticity Signal 10 (Rationale-Choice Expression Match) E10

Accounts whose games were all played before the promotion date never ran the extended classifier, so these columns are empty. This is exactly the missing (not contaminated) failure mode the measurement schema's SPOF finding calls out: the closure for missing central-extraction output is backfill.

Approach

A standalone one-time script that mirrors the live pipeline precisely:

  1. Select every player_decision_profiles row with an empty reasoning_style_distribution, excluding is_test_seed accounts.
  2. Pull rounds from choices (joined to sessions → scenarios) with created_at < 2026-04-12, in chronological order, with the stored rationale text. Speed rounds live in separate tables (speed_round_*) and never appear in choices, so the "skip speed_round" requirement is satisfied by construction.
  3. Classify each round's rationale through the same prompt + validation as src/app/api/ai/classify-driver/route.ts, on Sonnet 4.6 (MODEL_VERSION) — matching the live model is deliberate: a different model would contaminate the signal rather than fill the gap.
  4. Fold outputs into the three aggregates using the same EMA update functions as src/lib/ai/player-model.ts (updateReasoningStyleDistribution, updateAttentionDistribution, updateAuthenticity), with the same weighting (getUpdateWeight × game-type weight × 0.3 for NoTA).
  5. Replace the three columns wholesale and verify by re-reading.

Fidelity notes / known simplifications

  • NoTA rounds mirror the live sessions/complete path: rationale falls back to none_rationale, and the "closest option" is treated as the chosen one.
  • Game type (solo vs multiplayer, which sets the EMA weight) is inferred from session_participants count (>1 non-guest → multiplayer). This is a reconstruction; the live path knows it directly.
  • weightMultiplier for post-discussion Stand/Revise (multiplayer) is not reconstructed (default 1.0). It's a second-order scaling on EMA weight and doesn't change distribution shape materially.
  • E14 (ideal-actual reference frequency) and E18 (self-awareness) are out of scope. They are not classify-driver outputs — E14 needs a separate C2/C6 classifier that isn't in the current route, and E18 depends on milestone self-predictions (C7) that pre-promotion accounts never captured. They are listed as "indirectly affected" in the task but cannot be filled from classify-driver alone.

Idempotency & cost control

  • Classifications are cached to scripts/data/classify-driver-cache.json, keyed by sha256(promptVersion | scenarioId | optionId | rationale). Re-runs and resume-after-crash reuse cached results — zero extra API spend.
  • Columns are rebuilt and replaced (not folded into existing values), so a second --apply produces the same state. No double-counting.
  • Bounded concurrency (default 4) with 429/529 retry-backoff; cache is checkpointed every 10 classifications.

Dry-run results (2026-05-31)

npx tsx scripts/backfill-classify-driver.ts (no API calls, no writes):

8 accounts have an empty reasoning_style_distribution (non-test-seed). Breakdown:

Account Classifiable No-rationale Total pre-cutoff rounds
Amin (a75ff1dd) 14 0 14
Zer8 (a42d2067) 10 0 10
james (3ae483a5) 8 8 16
natascha.jasmin (79922238) 5 0 5
Test Player (49a17395) 17 0 17
skuhl (80a4631f) 0 0 0
tae+sync (3b5fca92) 0 0 0
ufkhan97 (3807a00c) 0 0 0
  • Classifiable rounds: 54 total (53 unique → 53 Claude calls)
  • Estimated cost: ~$0.38 (≈85k input / ≈8.5k output tokens, approx Sonnet rates)

Findings to flag before the live run

  1. The 3 confirmed teammates are all present and backfillable: Amin (14), Zer8 (10), James (8 of his 16 — the other 8 have no rationale text and are unclassifiable). Cici and CoachJ are correctly absent (their distributions are already populated).
  2. Auto-detect caught 5 more accounts. Two are real playtesters (natascha = Natascha, tae+sync = Tay from the RaidGuild playtest). "Test Player" (17 rounds) passed the is_test_seed filter — i.e. the column says it is not a seed — but the name warrants a human decision: include, or mark is_test_seed = true and exclude.
  3. 3 accounts (skuhl, tae+sync, ufkhan97) have 0 classifiable rounds. Their empty distribution is genuine absence of data, not a backfill gap. The live path skips them (no empty-over-empty write) and excludes them from the completeness denominator.

Results (live run, 2026-05-31)

Because only 53 rounds needed classifying, the classification step was done inline in-conversation (Claude is the same Sonnet-tier model the route calls) rather than via the API — zero API spend. The 53 classifications were written to the script's cache (scripts/data/classify-driver-cache.json), then --apply folded and wrote them with the deterministic EMA pipeline. The dump of the exact rounds classified is preserved at scripts/data/classify-driver-rounds.json.

5 of 5 backfillable accounts populated (100%). Verified by re-reading from the DB:

Account N reasoning_style.global (top) attention.global (top) authenticity (match)
Amin 14 consequentialist 50%, systems_thinking 45% financial 37%, interpersonal 26%, technical 25% 14 samples, 57%
Zer8 10 consequentialist 64%, eliminative 20% interpersonal 45%, ethical 32%, technical 23% 10 samples, 50%
james 8 consequentialist 63%, principled_deductive 17% interpersonal 86%, ethical 14% 8 samples, 50%
natascha 5 consequentialist 68%, eliminative 17% ethical 27%, interpersonal 25%, political 22% 5 samples, 40%
Test Player 17 principled_deductive 71%, intuitive 29% interpersonal 50%, ethical 50% 17 samples, 100%

All five also got per-category reasoning_style.by_category (governance / team-dynamics / values-culture / resource-allocation). last_updated stamped 2026-05-31 ~20:07 UTC.

Not backfilled (correctly): skuhl, tae+sync, ufkhan97 — 0 classifiable rounds, skipped (genuine data absence, not a gap). Cici + CoachJ were never selected (already populated).

⚠️ "Test Player" caveat — recommend marking is_test_seed = true. It populated, but the signal is thin/artificial: 10 of its 17 rounds had placeholder rationales ("yes yes yes", "test test test", "5168479564654564") which correctly classified to null reasoning mode / empty attention, and the remaining signal came from 6 identical "Test rationale from participant 2… aligns with my values around team harmony" rounds (hence 71% principled_deductive, 100% authenticity). This is test data wearing a real-account flag. If you mark it as a seed, re-running the backfill will simply skip it (the real five are unaffected — idempotent).

Re-running / idempotency

The columns are rebuilt-and-replaced and the classifications are cached, so re-running is a no-op on already-populated accounts (they're no longer selected once non-empty) and free on cache hits:

npx tsx scripts/backfill-classify-driver.ts            # dry-run / status
npx tsx scripts/backfill-classify-driver.ts --apply    # fold + write (0 API calls; all cached)

To redo an account from scratch, null its reasoning_style_distribution first, then --apply.