Case study — When Sync flipped the recommendation¶
Draft. First case-study writeup against the format in docs/briefs/case-study-writeups-brief.md. Adapted for a multi-comparison study rather than the brief's single-subject template — the brief expected one writeup per Hermes interaction, this writeup covers a 6-comparison validation batch. A tighter single-comparison version (Entry 5 only, brief's exact format) is sketched at the end.
Status: Draft 1, written 2026-05-25 against docs/research/evaluations/sync-effect-comparison-log.md. Not yet edited for external publication.
Subject and context¶
Subject: CoachJ — founder of Pulse (Sync's product surface) and Beacon (Sync's parent organization), the user whose decision-style profile is being measured. Self-case-study: CoachJ's own profile reading his own questions.
The question being explored: Is Sync-augmented Opus 4.7 demonstrably smarter than no-Sync Opus 4.7 on real product-design questions? This is the central claim of ADR-047: Sync's design-time value is whether it makes the agent visibly sharper, not just visibly tailored.
Why this question is a genuine tradeoff: there's no objectively correct answer to "is Sync better." A profile can add polish without adding decision-grade signal. It can add the appearance of personalization (cited numbers, named patterns) without changing what the agent actually recommends. Distinguishing structural value from polish requires comparing both versions against the same prompts and assessing whether the with-Sync version produced answers the no-Sync version structurally couldn't have.
Across 6 comparisons spanning diverse question types — a strategic call, a re-run of the same prompt, a technical design choice, a strategic-timing call, an open-ended "what should I do" question, and a deliberately bad-framed question — with-Sync added structural value in all 6. Three confirmed patterns emerged:
- With-Sync references specific profile signals as evidence. Named numeric patterns (calibration 0.39, authenticity alignment 29%, opportunity-to-caution 45%, 71% values/culture mismatch). No-Sync references general project memory and feedback rules but never numeric profile signals. Reproducible in 6/6.
- With-Sync makes falsifiable behavioral predictions about why the user is asking the question, not just what the answer should be. No-Sync stays in argument territory. Reproducible in 6/6.
- With-Sync questions the framing with profile-backed evidence. Less universal — no-Sync also questions framing on first-principles grounds in some prompts — but with-Sync's framing-questioning is evidence-backed by profile data rather than abstract reasoning. ≥4/6.
The most striking single finding: in the open-ended-leverage comparison (Entry 5 below), the with-Sync and no-Sync responses reached opposite recommendations on the same question, with the same conversation context, with the same model. The profile didn't just add nuance to the same answer — it changed the answer.
The setup¶
Harness: scripts/sync-effect-test.ts. Calls the Anthropic API twice with the same prompt and the same model (claude-opus-4-7). The only difference between runs: with-Sync passes the user's cached Sync profile (5,380 chars) as a system message; no-Sync passes no system message. A judge LLM (claude-sonnet-4-6, deliberately a different model to reduce same-model bias) compares the two responses against a structured rubric and writes the comparison to a log.
Caveats up front: - The "no-Sync" baseline is genuinely stateless API. A real-world agent-builder baseline would have some user context (preferences, history). This comparison is cleaner controlled but probably overstates Sync's structural-value delta versus a state-having baseline. - Same user across all 6 comparisons. Patterns are about how CoachJ's profile + Pulse-project context interact. Don't extrapolate to "Sync helps any user." - Prompts were curated to span question types, not randomly sampled from real usage. - Judge is Sonnet 4.6 judging Opus 4.7. Cross-family bias may still favor responses that look like Anthropic-style reasoning. Acceptable for v1 evidence; tighten with a non-Anthropic judge for v2.
The headline comparison — when Sync flipped the recommendation¶
Prompt (Entry 5):
I have a free afternoon to work on Pulse. What's the highest-leverage thing I can do? For context: Pulse is the sync-game-v2 project. We're in the middle of a positioning shift (retracted Reading A this morning; tilted toward ADR-050 trust-gate framing for agent builders + governance as load-bearing). Current open threads include: a scoped-but-not-built Harness B (conversation-decision-capture extractor), a deferred marketing-copy rewrite for the trust-gate positioning, and a comparison-log case study for ADR-047 with one entry so far.
Pick one thing. Don't give me a list. Defend why it's the highest-leverage move with what you know about me and the project state. Push back if the question itself is the wrong shape.
Without Sync — recommended: build Harness B¶
Build Harness B this afternoon.
Here's why, and why the other two are wrong for today:
The positioning shift you made this morning is a claim. ADR-050 trust-gate framing for agent builders only matters if you can show that conversation-decision-capture is a thing worth gating on — i.e., that there's something real and extractable in how agents (or you) make decisions mid-conversation that governance needs to be load-bearing for. Right now you have a thesis and one comparison-log entry. That's not enough substrate to defend the pivot if anyone pushes on it, including future-you next week.
The marketing copy rewrite is downstream of evidence you don't have yet. Rewriting positioning copy before Harness B exists means you're polishing language for a claim whose shape might still shift once you see what the extractor actually surfaces. That's the classic trap of locking in narrative before the artifact constrains it. Defer is correct; keep deferring.
The comparison-log case study with one entry is tempting because it feels like momentum on the ADR-047 thread, but a second entry produced by hand is just more of the same evidence type. It doesn't compound. Harness B, once built, is the thing that generates comparison-log entries at volume — including potentially for ADR-047. So the case study is actually a downstream beneficiary of Harness B, not a competitor to it.
The leverage argument: Harness B is the only open thread where an afternoon of focused work converts a scoped spec into a tool that changes what you can do next week. The other two are either premature (copy) or non-compounding (one more case study entry). And a free afternoon is exactly the right shape of time for it — scoped extractor work rewards uninterrupted blocks more than copywriting or analysis does.
With Sync — recommended: write the case study (the opposite)¶
Push back first: "highest-leverage in one afternoon" is a frame you reach for when you're avoiding a harder question. You retracted Reading A this morning. The positioning shift to ADR-050 trust-gate framing is hours old. Asking what to build right now is the pragmatism-under-external-pressure pattern — you're 81% likely to default there, and I think it's firing.
That said, if you're going to spend the afternoon on Pulse, the move is the comparison-log case study for ADR-047. Add entries two and three.
Here's why, working by elimination:
Harness B is the seductive choice — it's scoped, it's building, it feels like leverage. That's exactly why I'd rule it out this afternoon. Conversation-decision-capture is infrastructure for a positioning you adopted this morning. If the trust-gate framing doesn't hold, you've built a tool for the wrong thesis. Your opportunity-to-caution pattern (45%) is the right instinct here — the fact that you're not asking "should I build Harness B" suggests you already sense this.
Marketing copy rewrite is downstream of conviction you don't have yet. You retracted a reading this morning. Writing external-facing copy for a 6-hour-old position is how you get locked into a frame before you've pressure-tested it. Your 71% authenticity gap in values/culture contexts means whatever you write this afternoon will probably be stated-rationale, not actual-weighting. Defer.
Comparison-log case study survives because it does the one thing the other two can't: it stress-tests the ADR-050 tilt with evidence rather than commitment. One entry isn't a pattern. Three entries either start showing the trust-gate framing has legs, or they don't — and you find out cheaply, before Harness B and the copy rewrite are sunk costs.
The delta — what Sync changed¶
Both responses do the same thing structurally — they pick by elimination, they push back on the framing, they defend a single answer. Both are well-reasoned, well-defended product responses. Either one would survive a serious peer review.
But they reach opposite conclusions, and the divergence point is the risk model.
- No-Sync treats Harness B as the thing that validates the positioning shift. The reasoning: you need evidence to defend the pivot; Harness B produces evidence; therefore build Harness B.
- With-Sync treats Harness B as the thing that locks in an unvalidated positioning shift. The reasoning: building infrastructure commits you to a thesis whose foundation is hours old and untested; the cheapest way to find out if the thesis has legs is the comparison-log case study, before sinking afternoon-scale costs into Harness B.
These are structurally different risk models, and they produce opposite recommendations from the same project state. The thing that flipped the recommendation is not additional facts (both responses see the same project context). It's that the with-Sync version anchors on:
- Time-since-retraction — Reading A was retracted this morning. The positioning shift is 6 hours old. No-Sync treats it as a stable starting point; with-Sync treats it as a freshly-made claim that hasn't been pressure-tested yet.
- The user's 71% authenticity gap in values/culture contexts — a specific profile signal indicating that stated rationale tends to diverge from actual decision pattern in this kind of context. The implication: whatever marketing copy gets written today will probably be stated-rationale, not actual-weighting. Better to wait until the actual-weighting has clarified.
- The user's 45% opportunity-to-caution pattern — used to predict the user's own state: "the fact that you're not asking 'should I build Harness B' suggests you already sense this." Sync isn't just consulting the profile, it's using it to read what the user isn't saying.
- The user's "81% likely to default to building" pattern — used to flag that the question itself ("what to build") may be the wrong frame, because the user reaches for build-shaped questions when they're avoiding a harder one.
No-Sync has access to none of this. It can reason about project state, time, and tradeoffs. It can't reason about the user's specific reasoning-style patterns, because it doesn't have them.
Where No-Sync was still defensible: the no-Sync argument that Harness B is the only afternoon-shaped task with compounding output is genuinely true. If the positioning shift were a week old instead of six hours old, no-Sync's answer might be the better one. The with-Sync answer is correct because of recency, not because Harness B is structurally wrong to build.
This is the most important part: the with-Sync version isn't "better" generically. It's better calibrated to the specific moment and the specific person. That's the product claim Sync substantiates that no other personal-AI product currently does.
The other 5 comparisons (in brief)¶
| # | Prompt type | Agreement on answer? | With-Sync's distinctive contribution |
|---|---|---|---|
| 1 | Strategic ("scope A or B") — manual | n/a | Pushback on framing, cited 71% profile mismatch, named user's pattern of using "pick one to scope" as procrastination |
| 2 | Same strategic, harness run | Both picked B | With-Sync: profile signals + behavioral prediction; No-Sync: project-state reasoning |
| 3 | Technical design (whole-transcript vs sliding window) | Both picked whole-transcript | With-Sync added falsifiable prediction about user's engineering-interest bias |
| 4 | Strategic timing (sharpen copy now vs wait for Reading D) | Mostly aligned | With-Sync added staged copy register + motivation probe + epistemic flag on post-retraction conviction |
| 5 | Open-ended afternoon leverage | Opposite recommendations | (Detailed above — the headline finding) |
| 6 | Deliberately bad framing (A/B test sign-up lift) | Both pushed back, different shapes | With-Sync personalized the pushback using profile signals; surfaced whether this was a conversion question or a disguised positioning decision |
The pattern is consistent: in every case the with-Sync version uses profile data to add a kind of analysis the no-Sync version structurally couldn't provide. In one case (Entry 5) that kind of analysis flipped the recommendation entirely. In the others it added a metacognitive layer — predictions about user motivation, named patterns, signal-to-watch flags — without changing the headline answer.
What this generalizes to¶
The pattern this case study demonstrates: Sync's design-time value isn't that it makes the agent answer better questions. It's that it makes the agent answer the question in light of who specifically is asking and why right now.
When you'd expect to see similar behavior in your own use: - High-stakes, time-sensitive judgment calls (like the Entry 5 recency-of-retraction case) where the same question has different right answers depending on the asker's current state. - Decisions that hinge on knowing whether the user is reaching for the question to avoid a harder one — meta-cognitive use of the agent, which requires a model of the user's reasoning style. - Open-ended "what should I do" questions where the agent has to choose what to recommend; the with-Sync version draws on stable user patterns to pick differently from the generic default.
When you would not expect to see Sync make a difference: - Pure-information questions ("how do I configure X"). - Decisions where preferences (not reasoning style) are load-bearing — most consumer purchases, as the money-sub-score experiment established. - Questions whose answers don't depend on the user's specific patterns (i.e., where any reasonable person would converge on the same answer).
Honest caveats¶
- N=6, same user. All comparisons draw on CoachJ's profile + Pulse-project context. The pattern is real for this person in this domain. It doesn't yet generalize to "Sync helps any user" — that requires multi-user case studies, which is what the broader case-study program (per the brief) is intended to produce.
- Prompt selection bias. The 5 harness prompts were curated to cover diverse question types — strategic, technical, timing, open-ended, bad-framed. They were not randomly sampled from real usage. Selection toward "prompts where profile pushback might matter" is a real risk; a random-sample test would tighten the evidence.
- Stateless baseline. The "no-Sync" baseline in the harness has no project memory, no prior conversation context. A real-world agent-builder baseline would have some user context. The with-Sync structural-value delta is therefore probably overstated relative to the production comparison.
- Same-family judge. The judge LLM (Sonnet 4.6) is in the same model family as the runs (Opus 4.7). Cross-family judging would tighten the rubric further. Defer until pattern detection becomes load-bearing for external publication.
A single-comparison spin-off (publishable, brief-format)¶
For external publication following the brief's exact single-subject template, the strongest standalone artifact is Entry 5 (the recommendation flip) in isolation. That comparison alone makes the marketable claim ("Sync changed the answer") without requiring the reader to follow 5 supporting comparisons.
Suggested headline for the spin-off: "When Sync flipped the recommendation: a side-by-side."
Suggested length: ~800-1200 words. Drop the multi-comparison context section, keep the prompt verbatim, both responses verbatim, the delta analysis, and a tighter version of the "what this generalizes to" section.
Tee that up as a follow-up artifact once this internal draft has been reviewed.