Skip to content

ADR-070: Per-category sync scores — deferred pending category-axis decision

Status: Deferred Date: 2026-05-23 Context: CoachJ raised the per-category sync scores idea on 2026-05-21 (Slack 23:15): "We might be over-indexing on a single universal score, and could be good to design individual scores per category that make up the main score… if I wanted my agent to get really good at governance decisions right away, I could play lots of governance games, get its score really high in the governance category, but maybe its overall score isn't quite as high, and that's okay for what I want to accomplish."

A two-day design exploration (2026-05-21 → 2026-05-23) converged on a 5-category proposal (Money / People / Strategy / Dilemmas / Voice) with the audit/explanation reframe as a structural alternative, then was contested by deep-research findings on actual personal-AI usage patterns. Three blocking design questions remain unresolved; co-founder pushback raised the additional question of whether this work aligns with the light paper thesis at all.

This ADR formalizes the deferral so the next conversation isn't restarted from scratch.

At a glance

What it decides: don't build per-category sync scores yet. Deferred — revisit once the category-axis question (#1) is settled and the co-founder confirms light-paper fit (#3); #2 can be designed in parallel after #1.

  • Three blocking questions: which category axis, do scores do something or just visualize, and does this align with the light-paper thesis at all.
  • Why the 5-category v1 died: deep research showed "Strategy" and "Dilemmas" are designer projections that no user, study, or GPT actually names.
  • Live options: Menlo life-domains (validated but corpus-thin), keep the current 4 content categories (matches corpus, not user mental models), or the audit/explanation reframe.
  • Dark-horse: the audit/explanation surface keeps the specialization intent without committing to a scored axis the research says users haven't asked for.
  • Triggers to revisit: co-founder yes/no on light-paper fit, OR corpus gains enough Health/Logistics/Work scenarios, OR ADR-050 permission tiers need per-category trust.

Decision

Defer implementation of per-category sync scores until three blocking questions are resolved:

  1. Category axis: Adopt the research-validated Menlo life-domain taxonomy (Money / Health / Work-Career / Relationships / Logistics) and accept that several categories are data-thin from day one given the corpus mismatch, OR keep the org-leadership-flavored categories (governance / resource-allocation / team-dynamics / values-culture) that match the corpus but not how users actually segment their lives, OR take the deeper reframe entirely.

  2. Display vs. specialization: Are category scores cosmetic visualization, or do they actually do something? If purely visualization, they don't deliver CoachJ's original specialization intent. If they do something, the cheapest "something" is Hermes confidence calibration per category — but that needs to be designed.

  3. Light-paper alignment: Co-founder raised that this work is a divergence from the light paper thesis. Decision needed: is per-category scoring aligned with what Sync is trying to prove (governance threshold for AI delegation), or is it a GTM optimization that drifts from the core thesis?

Do not build until at least #1 and #3 are answered. #2 can be designed in parallel once #1 is resolved.

Rationale

Why the 5-category v1 didn't survive: Deep research grounded in NBER ChatGPT study (~1.1M conversations), Anthropic Economic Index, Menlo 5,031-person survey, GPT Store taxonomy, and ~16 user-shared agent configs showed that:

  • Strategy does not appear as a user-described category in any platform, study, or user-built GPT. Closest analog ("Business strategy & ops") is ~6% of Claude usage and lives inside Practical Guidance.
  • Dilemmas does not appear at all as a user-described category. Hard ethical questions exist as edge-cases inside Practical Guidance / Coaching clusters, never as a standalone category.
  • Money holds — Menlo flags it as a top underserved high-trust domain.
  • People holds but needs disambiguation (relationships? people-management? social/logistics?).
  • Voice survives but as a cross-cutting style overlay, not a peer category.
flowchart LR
    subgraph v1[5-category v1 proposal]
        Money
        People
        Strategy
        Dilemmas
        Voice
    end
    Money --> K1[Holds: top underserved domain]
    People --> K2[Holds, needs disambiguation]
    Strategy --> X1[Cut: designer projection]
    Dilemmas --> X2[Cut: designer projection]
    Voice --> V[Reframed: cross-cutting style overlay]

What the research did to each proposed category: two hold, two were cut as designer projections, and Voice survives only as a style overlay.

The Menlo life-domain taxonomy (Money / Health / Work / Relationships / Logistics) is better-validated but has the corpus mismatch problem: the existing scenario corpus is org-leadership-flavored. Almost no Health scenarios, no Logistics scenarios. Adopting the validated taxonomy means either substantial corpus rebuild or accepting several categories are functionally ungrindable.

Why CoachJ's profile complicates the decision-making: Reading his actual decision profile (fetched 2026-05-23) — 67% overconfidence rate, 70% consensus deviation, 29% authenticity in values-culture decisions, opportunity → caution at 45% — suggests his rapid agreement to design conclusions in this domain should be probed more, not converged on. His enthusiasm for the 5-category proposal pattern-matches to high-conviction-low-accuracy.

Why the audit/explanation reframe is the dark-horse option: Instead of fixed categories users can grind, surface qualitative patterns the profile has revealed ("Across 14 money decisions, you've consistently picked downside protection over expected value upside") with linked evidence. Preserves the original specialization intent (users see where they're strong, agent leans into it) without committing to a product shape the research says users haven't reached for. Avoids the corpus mismatch problem entirely. The deferred answer to question #2 above.

Alternatives Considered

  • Ship the 5-category v1 (Money / People / Strategy / Dilemmas / Voice) as designed: Rejected by the research. Strategy and Dilemmas are designer projections.

  • Adopt Menlo life-domain taxonomy directly (Money / Health / Work / Relationships / Logistics): Open. Requires corpus expansion or acceptance of thin data per category. Has external research validation but disrupts existing scenario corpus alignment.

  • Keep current 4 content categories (governance / resource-allocation / team-dynamics / values-culture) and expose them as scores: Open. Matches corpus exactly; doesn't match user mental models of "what I want my agent good at"; doesn't deliver CoachJ's specialization intent for non-org-leadership users.

  • Replace with audit/explanation surface (no fixed categories): Open. Patterns emerge from play; universal sync_score stays primary; visualization is qualitative cards with evidence, not scored axes. Dodges corpus mismatch and aspirational-gap problems.

  • Ship as pure visualization with no downstream effect: Rejected. CoachJ explicitly named the requirement that categories do something ("if it's just pure visualization then it's meaningless").

Discussion

This decision is best understood as the product of a two-day design conversation that produced strong convergence and then strong reversal. The reversal came from three sources, all valid:

  1. Empirical research showed two of five categories were designer projections
  2. CoachJ's own profile data suggested his agreement-speed was suspicious (high overconfidence + low values-culture authenticity)
  3. Co-founder pushback raised the light-paper alignment question that hadn't been considered

Worth preserving the deliberation context: when the categories conversation started, there was real product enthusiasm and a clear path (5 categories → radar UI → Hermes calibration). By the end, the path had collapsed and was replaced by three blocking questions. That's not a failed conversation — it's a successful one, because the cost of shipping the wrong axes (corpus rework, UX iteration, downstream gating decisions on category trust) is much higher than the cost of another design pass.

The audit/explanation reframe is genuinely the most interesting alternative because it captures the intent of the original idea (specialization, transparency, agent leans into strengths) without committing to a product shape that the research says users haven't asked for. It's also the option that's most aligned with what Sync uniquely measures (decision-style signals, not domain expertise).

Adjacent decisions made in the same conversation that should inform any future work here:

  • ADR-065 (Behavioral attention signals): Proposed surfacing political attention, temporal attention, calibration, consensus deviation, and reasoning style as named user-visible dimensions. This is the complement to per-category scores — categories expose domain depth, behavioral signals expose processing style. Either can ship without the other.

  • CoachJ profile as live context (memory: user_coachj_sync_profile.md): Fetching the actual profile during design conversations would have changed several recommendations earlier in the categories arc. Future sessions should refresh this as games are played.

  • The autonomous-twin vision: CoachJ articulated that the categories are ultimately permission tiers for an autonomous agent that self-queues product work. If that vision becomes load-bearing, per-category scores have a different design constraint (they map to action classes the agent can act on alone) than the user-facing "what should my agent be good at" framing. ADR-050 (deferred permission tiers) is the governance layer this would require.

Consequences

  • No category scores will ship until questions 1 and 3 are resolved
  • The category_patterns data in player_decision_profiles (already measured, surfaced on insights page) remains the only category-axis surface
  • Trigger conditions to revisit: (a) co-founder alignment conversation produces a clear yes/no on light-paper fit, OR (b) corpus expansion lands enough Health/Logistics/Work scenarios to make Menlo-axis viable, OR (c) ADR-050 permission tiers ship and require per-category trust as their primitive
  • Memory project_category_sync_scores.md remains the working design document; this ADR is the formal status

Key files: - memory/project_category_sync_scores.md — full design history and current state - docs/research/briefs/openclaw-hermes-research.md — the research artifact that contested the v1 design - docs/decisions/065-behavioral-attention-signals.md — complementary surface (processing style, not domain depth) - docs/decisions/050-agent-permission-tiers.md — the deferred governance layer that would require per-category trust as a primitive - docs/research/audits/conversation-decisions-audit.md — source items 74, 75, 76, 77