ADR-047: Defer SKILL.md instruction-tuning until team-usage feedback signal¶

Status: Accepted (the deferral played out — signal received May 9; see Outcome) Date: 2026-05-08 Context: The Sync agent skill v2.0.0 (May 8, 2026) — public/agent-skill/SKILL.md — replaced the v1 raw-curl approach with first-class MCP tool references. The v1 → v2 rewrite fixed two real bugs uncovered in CoachJ's first end-to-end functional test:

v1 SKILL.md tripped Hermes' security scanner (DANGEROUS / CRITICAL exfiltration verdict on the curl-with-Bearer patterns) → distribution blocker for any user without --force.
v1 SKILL.md was treated as flavor-text by the model: agent answered with confident-sounding position but never called the Sync API, fabricating patterns from session memory instead.

v2 fixed both: scanner verdict went to SAFE, agent now calls sync.get_recent_decisions and grounds recommendations in real past decisions. The infrastructure-side win is the F1 / chief-of-staff thesis working in real life — case-study material.

But the v2 functional test (May 8, post-rewrite, Telegram + Hermes + Haiku 4.5) surfaced four polish gaps in the agent's response:

~280 words against a 120-word cap. SKILL.md hard rule §2 not followed.
No Was this useful? 👍 👎 feedback prompt at the end. SKILL.md feedback section not followed.
Embodiment is partial. Agent says "Based on your patterns, you'd unblock James" — better than narration, but still 3rd-person ("you'd") rather than first-person ("I'd lean toward unblocking James"). SKILL.md hard rule §3 not followed.
Called get_recent_decisions first, skipped get_profile. SKILL.md tools-section says get_profile is the load-bearing first call.

These are all instruction-following gaps — the rules are in the SKILL.md text, the model isn't following them tightly. We have a choice about whether to react now.

At a glance

What it decides: Hold off on tuning SKILL.md for four observed polish gaps until there is ≥1 week of real team-usage feedback — don't speculatively tighten language off a single test transcript. (The deferral resolved fast: signal arrived May 9.)

Tuning twice is worse than tuning once with data — the four gaps (verbosity over the 120-word cap, missing feedback prompt, 3rd-person voice, tool-call ordering) are real but it's unknown which actually matter to users.
The feedback loop is the feedstock — sync.submit_feedback plus the eval harness will map 👎s to specific recommendations; tuning before that loop closes is guessing.
Some gaps may be model, not instructions — Haiku 4.5 relaxes long-tail rules under pressure; test across Sonnet/Opus first, tune only if needed.
Outcome (May 9): Cici's N=1-but-decisive signal confirmed gap #1 — short answer first, "why" on demand; the SKILL.md rewrite then proceeded as separate implementation work.

Decision¶

Defer SKILL.md instruction tuning until we have ≥1 week of team-usage feedback data. Don't speculatively tighten language for the four polish gaps above based on a single-test transcript. Capture the gaps here, set explicit triggers for revisit, and let real-world signal drive the prioritization.

Rationale¶

Three reasons:

Speculative tuning twice is worse than tuning once with data. The four observed gaps are real, but we don't know which actually matter to humans. A user might love a 280-word grounded answer with no feedback prompt. Or they might hate the verbosity but never notice the missing prompt. Or they might find the 3rd-person framing fine. Tuning now without that signal means we'll likely re-tune after team feedback anyway — two iterations, both partial.
The thumbs-up/down feedback loop is itself the data feedstock for tuning. The MCP tool sync.submit_feedback exists, the read_id round-trip works, the eval harness already tracks per-choice agreement under three context conditions (src/lib/agent/eval.ts). When team users actually use the skill and react, we'll have structured signal mapped to specific recommendations: "these answers got 👎, what do they share? are they too long? did they fabricate? did they pick the wrong category?" That's how you tune SKILL.md properly. Tuning before the loop closes is guessing.
Some of the gaps may not be SKILL.md problems at all. Haiku 4.5 is a small model; it relaxes long-tail rules under pressure to be helpful. The same SKILL.md may follow rules tightly under Sonnet 4.6 or Opus 4.7. Tightening the instructions before testing across models means we might over-fit to Haiku's failure modes and hurt the larger-model behavior. We should test across models first, then tune if needed.

Alternatives Considered¶

Tighten SKILL.md immediately. Make the 120-word cap a hard count, add ALL-CAPS imperative language ("ALWAYS call sync.get_profile FIRST"), restructure the feedback prompt section to lead the response template. Rejected as primary path — this is the speculative-tuning-twice trap. Worth doing later if the data confirms specific gaps. Cost-of-not-doing-now is small: a few responses with rough edges while team users adapt.
Test the same prompt against Sonnet 4.6 / Opus 4.7 before tuning. Strongly considered. This is the single highest-leverage diagnostic available — if Sonnet nails all four rules unchanged, the issue is model-not-instructions. Recommended as the first step when revisiting; it's not deferred indefinitely, just out-of-scope for this push. CoachJ's Hermes is currently on Haiku 4.5 by default; switching is a config change.
Add a few-shot example to SKILL.md showing a perfect 120-word response with feedback prompt. Considered. Few-shot is known to help small-model rule-following. Rejected for now: same speculative-tuning argument, plus risk of the example becoming the literal template the model parrots, reducing variety.
Move the feedback-prompt firing to a server-side enforcement (e.g., the SKILL.md or MCP layer always appends "Was this useful?" to recommendations). Rejected: defeats the point — the prompt is meant to fire when the agent did consult the profile, not on every response. Server-side enforcement would fire on every call and produce noise.

Discussion¶

The user asked the right question on May 8: "What do you think we should do here? Should we go ahead and tighten and get the thumbs up, thumbs down working, and then move on to more infrastructure stuff? Or should we just leave it at that and call that a win and come back to that later as one of the ADRs that we wrote up?"

The answer is the second path because the v2 rewrite already shipped the load-bearing win — the agent calls real tools and grounds recommendations in real past decisions. That's the chief-of-staff thesis working. Polish gaps don't move the GTM-relevant case study materially.

Two specific framings tipped the call:

The eval harness is already running. Every choice in Sync writes to evaluation_predictions under three conditions (A: personal_context only, B: profile only, C: both). The infrastructure for measuring agent-with-Sync vs agent-without-Sync exists. The thing that's been missing — structured user feedback on real-world recommendations — is what sync.submit_feedback provides, and that fires from team usage. Closing that loop is the precondition for principled tuning.
The case study is built on what works, not what's polished. The May 8 v2 transcript already shows: agent calls a real Sync tool, names two specific past scenarios from the user's gameplay (The Lifeboat Equation, The Floor Plan Map), takes a position, attributes the recommendation to a real pattern. That's the demo material. Length-cap and feedback-prompt fixes are tunable; they don't unlock new product capability.

CoachJ's parallel priority — L4 scaffold + Cici's skill lock+rollback — is also stronger signal-generation work than SKILL.md tuning. Cici will use the lock feature; her usage tells us what real users care about. Speculative SKILL.md tweaks have no comparable signal source until the team is using it.

Consequences¶

The four polish gaps remain in observed behavior. Heads-up for early team users (Cici, James, Tash, Amin, Natascha) is in the install guide (Behavior expectations section added May 8) and in the team rollout Telegram message.
The sync.submit_feedback MCP tool stays in place. Agents that do fire the feedback prompt log structured signal. Agents that don't (Haiku occasionally) miss the loop entirely — accepted cost.
Tuning, when it happens, will be data-driven. No more "this looked rough in a single test" tuning. Look at aggregate patterns in agent_response_feedback + evaluation_predictions and tune to what's actually moving 👎s or low confidence.
Cross-model behavior testing is now a real workstream (probably an ADR-049 or similar when it triggers). The single biggest open question is whether Haiku-specific tightening hurts Sonnet/Opus behavior. Need to test before deciding.

Triggers for revisit (any one):

≥1 week of team usage with ≥10 recommendations each from ≥2 users — enough to have aggregate feedback signal. Then look at the data: which gaps are real, which are noise.
A specific gap repeatedly draws explicit complaint. "Your agent gives me a wall of text every time" or "the feedback prompt never appears" said by ≥2 different users → that gap is the priority, regardless of aggregate stats.
Cross-model test produces cleaner Sonnet/Opus output with the current SKILL.md → tuning becomes optional / model-routing becomes the answer instead.
A specific high-stakes user adoption moment (Ryan Sarver, a pilot like Lena/Capsule, a Metagov / Raid Guild demo) where polish would materially affect adoption. Then we tune for the demo with explicit before/after, but only that situation justifies pre-emptive tuning.

Key files: - public/agent-skill/SKILL.md — the file that would be tuned - src/lib/agent/submit-feedback.ts + src/app/api/agent/v1/feedback/route.ts — feedback ingestion path - migrations/081_agent_eval_harness.sql — evaluation_predictions table; the offline measurement instrument - migrations/084_agent_feedback.sql — agent_response_feedback table; the structured user signal table - src/app/admin/agent-eval/page.tsx — the dashboard that should grow a "feedback breakdown" panel when this ADR's triggers fire

Outcome — signal received May 9, 2026¶

Trigger fired earlier than expected. On the May 9 cici×coachj 1:1, Cici's first reaction to the v2 functional-test transcript:

"The amount of tokens you're spending on getting all this back is potentially just a waste. If you just want an answer, I would want the agent to just respond to the answer, and then I want the option to say why, because the output tokens — all that writing there — is more expensive than the input tokens."

This is direct signal on gap #1 (verbosity, ~280 words against a 120-word cap). Doesn't quite meet the original "≥2 different users repeatedly complain" bar (only N=1 user), but for a 5-person team, N=1 is meaningfully large — and the framing is unambiguous: short answer first, "why" on demand, not the other way around. The other three gaps (no feedback prompt, 3rd-person, profile-vs-recent-decisions ordering) remain unaddressed by this signal.

Direction confirmed for the SKILL.md rewrite: - Default response shape: decisive recommendation in 1–2 sentences, then a one-line "Reply 'why' for the reasoning" tail. Don't dump the full reasoning in the first response. - Reasoning lives behind a follow-up trigger word, not in the headline. - Pattern that works (per Cici): "Recommendation: X. (One-line justification.) Reply 'why' for the full reasoning." - Watch token budget per query in Hermes telemetry — if median output tokens exceed ~200, the SKILL.md is still asking for too much. - Same UX shape as the Sync (Play) recommendation hero card per ADR-045 v2 — short headline, expand on tap.

Implementation note: the actual SKILL.md rewrite + cross-model testing is being driven by Jonathan in a separate Claude Code chat (May 9+) since it requires substantial back-and-forth and Hermes-side iteration. This ADR's job — codifying the deferral decision and documenting when/why the signal arrived — is done. The SKILL.md tuning itself is implementation work, not architectural.

Memory: see feedback_skill_md_concise_first for the canonical user-feedback capture.