Skip to content

Experiment — Conversation-as-Decision-Capture

Status: Scoped, not yet started Opened: 2026-05-25 Origin: Surfaced by CoachJ at session close of the judgment-vs-normative-competence arc; see also the session note — outside repo. Connects to: ADR-039 (non-game decision capture — missing primitive), conversation-decisions-audit (the retroactive 88-item pass that proved the format), experiment-money-subscore.md (Reading D recommendation #4 — adjacent in reverse).

Why this experiment exists

CoachJ surfaced an observation at the close of the 2026-05-25 judgment-arc session: throughout that conversation he made dozens of real governance-shaped tradeoffs (Reading A vs B vs C; close experiment at N=1 vs N=15; one money sub-score vs seven; defer marketing rewrites vs push through; how to pitch agent-builders; etc.). All exactly the shape Sync claims to measure. None of them are being captured — Sync only ingests scripted-scenario gameplay.

The conversation-decisions-audit two days ago (2026-05-23) was the retroactive version: a 6-month pass over Claude Code history that produced 88 documented decisions and seeded ADRs 066–071. It proved the extraction shape works. But retroactive only fires when CoachJ remembers to do it; the value is in prospective, habitual capture.

This experiment tests whether automated post-conversation extraction can produce real-world decision data of comparable quality to the retroactive audit, at low enough friction to become routine.

The hypothesis

An automated post-conversation pass (run at session close, against a Claude Code transcript) can extract major tradeoffs from a session — options considered, option chosen, rationale — into the same shape the conversation-decisions-audit produced manually. At quality comparable to manual extraction. At low enough friction that it gets run every session.

If true: Sync gains a steady stream of real-world reasoning-style decisions, addressing the "Sync only knows your gameplay" gap that's been hovering under multiple research threads. If false: the format that worked retroactively doesn't translate to automated extraction, and the missing primitive (ADR-039) still needs a different solution.

The three candidate approaches (and why this experiment picks one)

Three approaches were sketched at session close:

  1. Post-conversation extraction — automated pass at session close pulls major tradeoffs out of the transcript. (This experiment's pick.)
  2. In-conversation tagging — explicit /decision skill captures decision moments as they happen. Connects to Cici's skill-lock idea.
  3. Weekly review — Friday/Monday structured prompt asking for 3-5 biggest decisions of the week.

This experiment picks approach #1 because:

  • Lowest friction. Runs automatically; no new behavior required from the user.
  • Highest catch rate. Doesn't depend on remembering to tag in-the-moment.
  • Format proven. The conversation-decisions-audit already validated the extraction shape — we're just automating what was manual.
  • Free ongoing value. Once wired into session-close, every session contributes data with no marginal effort.

The other two approaches aren't ruled out — they could complement #1 (e.g., in-conversation tagging for moments the extractor would miss). But #1 is the cheapest test of the central claim.

Why this experiment before Reading D

The judgment-vs-normative-competence doc also recommends scoping Reading D — Sync as live-context introspection where an agent asks the user 1–3 calibrated questions at decision time. Both Reading D and this experiment respond to the same underlying gap (Sync only knows your gameplay, not your real decisions). They look like alternatives. They aren't — they're sequentially dependent, and this one comes first.

The sequential-dependency argument. Reading D requires a corpus of real decisions the user has actually made, with context, to be testable. If you build a "live-context introspection agent" and test it on synthetic decisions, you'll get synthetic answers — same failure mode the money-sub-score experiment hit at N=1 (manufactured scenarios elicit preferences, not behavior). To run Reading D well, you need exactly what this experiment produces: a growing log of real-world governance-shaped tradeoffs with context. B → A is a clean experimental pipeline. A → B is a prototype in search of evaluation data.

The asymmetric-bet argument. B produces durable data; A produces a prototype. Run Reading D and fail — you have nothing left over. Run B and fail — you still have months of real-world tradeoffs logged against the profile, addressing the ADR-039 missing primitive independent of what Reading D turns into. This is the direction the project's standing feedback rules (ship-and-observe, no API spend when inline works) already favor.

The doc-level recommendation: scope and run B first. Treat Reading D as work that comes after B has produced a usable decision corpus.

Setup

Input

  • One Claude Code session transcript (full conversation history, both sides).
  • The session-close moment as trigger.

Output

  • Zero or more decision entries in the same shape as docs/research/audits/conversation-decisions-audit.md:
  • Decision title (1 line)
  • Options considered (bulleted, with brief rationale per option)
  • Option chosen
  • Rationale
  • Optional: tags (which category, which axes, which Sync signals it would inform)

The acid test

A decision entry is "valid" if it would survive review by CoachJ. The simplest test: run the extractor on this very session (the judgment-arc session that produced this experiment), compare the extracted decisions to the actual decisions made in the session. Did it catch Reading A vs B vs C? Did it catch close-at-N=1 vs N=15? Did it catch the marketing-rewrite-deferral?

Quality bar (operationalized)

  • Recall ≥ 60% of decisions a human reviewer would name as "major" in the session.
  • Precision ≥ 70% — at most 30% of extracted entries are noise (false positives, trivial choices, or restatements).
  • Friction: must run in ≤ 30 seconds at session close, must not require any new user input beyond a yes/no confirmation.

These thresholds are first-pass — calibrate after the first real run.

Method

  1. Define the extraction prompt. A single prompt that takes a transcript and returns a JSON list of decision entries. Modeled on the conversation-decisions-audit format. Inline in the experiment doc until proven.
  2. Wire to session-close. Either as a slash command (/extract-decisions) the user runs manually at close, or as a hook tied to the existing session-close ritual. Start with manual to avoid scope creep.
  3. Run against this session. The judgment-arc session is the first test case — known-good ground truth (CoachJ remembers what he decided).
  4. Per-decision review. No-bulk-audit policy applies. CoachJ eyeballs each extracted entry, marks valid/invalid/needs-edit.
  5. Compute recall + precision. Against CoachJ's own list of decisions from the session.
  6. If quality bar met: scope integration into the Sync profile — how do extracted decisions become signal-bearing entries? Open question that depends on what the extractor actually produces.
  7. If quality bar not met: diagnose. Is the prompt wrong? Is the extraction format wrong? Is automation the wrong move (back to in-conversation tagging)?

Results

To be filled in as the experiment runs.

What we learned

To be filled in at experiment close.

Open questions worth holding in mind

  • Where do extracted decisions live? A new table in Supabase? A markdown file per session? Both? The conversation-decisions-audit lives in docs/research/; that pattern might extend.
  • How do extracted decisions become Sync signal? They're not gameplay scenarios — they don't have predicted outcomes or driver classifications. Integration with the existing 12-signals pipeline is non-trivial. Possibly a separate "real-world decisions" track that complements rather than feeds gameplay signal. Defer until extractor quality is known.
  • Does this only work for Claude Code sessions? The trigger and the transcript shape are CC-specific. If the pattern works, the natural extension is any structured conversation — Slack threads, design docs, meeting transcripts. But start with CC because the friction is lowest (transcript is already there, session close already exists).
  • Privacy implications. Extraction surfaces and stores decisions. If anything sensitive comes up, the extraction layer needs filters. Out of scope for v1 but flag it.

Next steps if this experiment validates

  1. Wire /extract-decisions (or equivalent) into the project skills.
  2. Open a real ADR scoping the integration with the Sync profile.
  3. Consider whether to extend beyond Claude Code (Slack, meeting transcripts).
  4. Update ADR-039 (non-game decision capture missing primitive) — possibly supersede with "post-conversation extraction" as the concrete implementation.

Next steps if it falls flat

  1. Pivot to in-conversation tagging (/decision skill) as the alternative.
  2. Or accept that automated extraction at quality is harder than the conversation-decisions-audit made it look, and document why.
  3. The retroactive audit pattern still has value — keep it as the manual path.