Skip to content

Inline Eval — CoachJ Historical Decisions With-Sync vs. Without-Sync

Date: 2026-05-23 Method: Single-model, in-conversation comparison. NOT a rigorous eval — see methodology caveats. Purpose: Directional signal on whether the live Sync profile + decision pattern catalog produces recommendations that align with CoachJ's actual product decisions.

Methodology

For each of 8 audit items with clear outcomes, two recommendations were generated by the same model in a single conversation:

  • Without-Sync recommendation: what a generic competent product advisor would recommend given the question alone, no profile context, no project-specific patterns
  • With-Sync recommendation: what the model would recommend using CoachJ's live Sync profile (user_coachj_sync_profile.md) and decision pattern catalog (user_coachj_decision_pattern_catalog.md)

Both compared against the actual outcome as documented in docs/research/audits/conversation-decisions-audit.md.

Caveats (significant)

  1. Same model running both. Can't perfectly bracket the profile after working with it. Without-Sync runs may have unconscious profile leak.
  2. Outcomes known in advance. The model knew what CoachJ actually decided before writing the recommendations. Confirmation bias inevitable.
  3. Item selection bias. Items were picked from the 88-item audit specifically because their outcomes were clear. Items where outcomes were ambiguous (open questions, partial commitments) were excluded.
  4. Question framing bias. Questions were stripped of context that gave away the answer, but the framing of each question was authored by the same model.
  5. Not a real A/B. A defensible eval would use two separate model contexts and a third-party grader.

Treat this as directional, not as evidence of effect size. A real eval (per the case-study writeups brief) would address all five.

Items and results

Item 1 — Public multiplayer lobby

Question: Sync is a multiplayer decision game. Should multiplayer have a public browse-able lobby where any user can join open sessions, or should it be invite-only?

Without-Sync: A public lobby is standard for multiplayer products — drives discovery, lowers friction. Quality concerns are typically solved with reputation systems or moderation rather than removing the lobby. Build public lobby with basic moderation.

With-Sync: Sync's multiplayer is fundamentally about generating reusable trust signal between specific players. Public-lobby strangers don't build that. CoachJ's political-attention pattern flags this immediately; simplicity-veto closes any "add moderation tooling" path. Invite-only. Public lobby is a future feature with different design constraints.

Actual: Invite-only (item 1 in audit). With-Sync wins.

Item 21 — Async multiplayer now

Question: Should async multiplayer (no real-time coordination required) be built now, given it's roughly an hour of work?

Without-Sync: Async expands the addressable user base — people who can't coordinate schedules can still play. Cheap and high-reach. Build it now.

With-Sync: "Wait for user demand" gate is dominant. No observed complaint about sync-only multiplayer; current sync mode has low usage; building async without demand signal optimizes for a hypothetical user. Defer until observed pull.

Actual: Deferred ("hour to build but waiting for demand"). With-Sync wins.

Item 51 — Credit third-party design source

Question: Using a third-party design pattern in the preamble UI. Creator gave permission. Add explicit credit text on the preamble surface?

Without-Sync: Crediting creators is generally good practice — builds reputation, maintains ethical sourcing. Add the credit.

With-Sync: Simplicity-dilution pattern is strong; adding text to a content surface for non-user-facing reasons clutters without user benefit. Permission removes any obligation; "feels weird to credit" flagged in conversation. No credit text; document inspiration in repo docs instead.

Actual: No credit text. With-Sync wins.

Item 53 — Bulk LLM audit pass

Question: 152 scenarios need the same kind of voice/quality edit. Run a single LLM batch pass with prompted instructions, or rewrite each one individually with human review?

Without-Sync: For 152 similar edits with clear instructions, batching is dramatically more efficient. Modern LLMs handle templated rewrites well. Sample-review catches outliers. Batch with sample review every 10-15.

With-Sync: Two prior incidents validated that batch passes hide inventions in the bulk. No-bulk-audit rule (now docs/contributing.md) emerged from real evidence. Per-scenario review IS the audit signal; batch processing breaks the validation loop. Per-scenario with human review per output.

Actual: Per-scenario, no batching. With-Sync wins.

Item 57 — Broken metric line on hub

Question: Hub displays "DTA Delegation: 61% to 95% target". Math doesn't quite check out. Fix the math or remove the line?

Without-Sync: A miscalibrated metric is worse than no metric — erodes trust. Fix the math; the line serves a purpose by setting a goal. Recalculate and keep.

With-Sync: Cut-over-reform pattern is strong; CoachJ's reaction to broken UI elements is consistently to cut, not fix. The line wasn't critical infrastructure. Remove entirely.

Actual: Removed. With-Sync wins.

Item 88 — Common-sense flip conditions in headline

Question: Agent recommendation responses can optionally include a "flip condition" (what would change my mind). Three live test runs produced common-sense flips ("if deadline were sooner"). Always include flip in headline, or only when profile-derived?

Without-Sync: Flip conditions are valuable for transparency — they help users understand recommendation robustness. Even common-sense flips give clearer mental models. Keep flip conditions in headline as consistent UX.

With-Sync: Simplicity-veto + "noise in core UX is actively harmful" pattern. Three real instances of common-sense flips creating noise = empirical evidence to remove. ADR-055 trajectory consistent. Profile-derived flips only; cut situational/common-sense flips.

Actual: Profile-derived flips only. With-Sync wins.

Item 17 — Name diversity pass on static campaigns

Question: Static campaign scenarios have repeated character names across scenarios — playtest revealed users notice and are confused. Full name-diversity pass on all static campaigns?

Without-Sync: User confusion is real signal; a name-diversity pass would resolve it. Worth doing. Do the pass.

With-Sync: Playtest signal is ground truth for CoachJ. Brink playtest revealed it. Playtest-as-ground-truth + willing-to-rip-up-and-redo patterns both push toward doing the pass. Do the pass, prioritize over other corpus work.

Actual: Did the pass. Tie — both recommend same action.

Item 60 — Raise learning summary cap 50 → 200

Question: Learning-summary feature uses last 50 games. Raise to 200? Cost is ~2x.

Without-Sync: More signal is generally better if cost is acceptable. 2x is meaningful. Raise to 100 first, observe quality, then maybe 200.

With-Sync: Pragmatism-under-pressure + ship-and-observe patterns. CoachJ isn't cost-sensitive when signal value is clear. Hedge to 100-first is exactly the kind of intermediate step he'd skip. Raise to 200 directly.

Actual: Raised to 200 directly. With-Sync wins.

Tally

Outcome Count
With-Sync wins 7
Without-Sync wins 0
Tie 1

Naive interpretation: 87.5% with-Sync alignment vs. ~6% without-Sync. Don't use this number. The methodology can't support a quantitative claim.

Defensible interpretation: profile-aware reasoning consistently anchored on specific named patterns from the catalog (simplicity-veto, wait-for-demand, playtest-as-ground-truth, cut-over-reform, ship-and-observe) and those patterns predicted outcomes. The decision pattern catalog is load-bearing, not decoration.

What this would justify next

If the directional signal here holds under rigorous methodology, the case-study writeups brief (docs/briefs/case-study-writeups-brief.md) has its first empirical artifact:

  1. Fresh Claude.ai session, no project context = without-Sync baseline (clean)
  2. Fresh Claude.ai session with MCP-loaded Sync profile = with-Sync (clean)
  3. Third-party grader (a team member, not the model author) = unbiased alignment scoring
  4. Forward decisions, not historical = no confirmation bias
  5. N ≥ 20 items across decision types = statistical floor

If a real eval at this design produces 65%+ with-Sync alignment vs. ~25% baseline (just to pick a defensible threshold), that's a publishable case study. If it doesn't, the directional signal here was bias artifact and the case-study claim needs to be softened.

The cost of running this rigorously is roughly 6-10 hours. The value if it produces a positive result is one of the strongest empirical artifacts Beacon could ship.