Categorical classification for the artefact-resonance judge

At a glance

• When a new artefact is imported, an LLM “good-neighbour” judge scores how much each existing thread would be enriched by it, and links above a keep floor (0.4) are surfaced.
• A freshly imported image fragment surfaced many resonances but dropped a genuinely oblique cross-domain thread — the judge scored it 0.28, below the floor. Evidence
• Root cause, measured: the judge scores all candidates for an artefact in one model call. Scored alone, the oblique thread got 0.72; batched among ~33 candidates (the production path), 0.227. Evidence
• The freehand magnitude was collapsing into “how does this rank against the others,” so an oblique link lost to on-topic competitors sharing the call.
• Two intuitive fixes failed empirically: capping the batch (the thread stayed below the floor at every cap) and scoring each thread in isolation (recovers it, but then unrelated threads also score 0.57–0.72 — no discrimination left). Evidence
• Prompt engineering fixed the judge’s prose but not its number (mean 0.38, high run-to-run variance). The magnitude was the weak link.
• Decision: stop asking for a number. Classify the kind of relationship from a fixed vocabulary, then map each kind to a fixed score. “Is this a complication or a cross-domain grounding?” is stable under batching in a way “rank these 0–1” is not.
• Two kinds map below the keep floor, three above it — so the score column, the floor, and the surface are all unchanged. A drop-in replacement.
• The deliberately-unsolved part: with many genuine resonances, how many to show is a surface/ranking question, not a reason to cut good links in the judge.

The problem

Pond surfaces resonances: when you add something to your library, it points out which of your existing threads that artefact would enrich — especially the oblique, cross-domain connections you wouldn’t go looking for.

The mechanism is an LLM judge. For a newly imported artefact it looks at the candidate threads and decides which are worth surfacing, keeping anything above a fixed score floor. The judge’s whole job is to tell a real, enriching traversal apart from an incidental keyword overlap.

The trigger was a concrete miss. A newly imported image fragment surfaced a healthy set of resonances but dropped one thread that was exactly the kind of oblique cross-domain link the product is built to find. The judge had scored it 0.28, under the 0.4 floor, so it never showed. Evidence The question was why a connection a human reader would call apt scored as near-noise.

Decision map

Timeline

1. Jun 2026 · the trigger
   An oblique cross-domain link is dropped Evidence

   A newly imported image fragment surfaced many resonances but dropped a thread that a reader would call an apt, oblique connection. The judge had scored it 0.28, under the 0.4 keep floor.

2. Jun 2026 · root cause
   Isolated vs batched measurement Evidence

   Scored on its own against the artefact, the thread got 0.72 — the judge clearly sees the link. Scored in the batch of ~33 candidates (the production path), 0.227, matching production’s 0.28. A freehand 0–1 magnitude asked for all candidates at once collapses into relative ranking: the oblique link loses to on-topic competitors in the same call.

3. Jun 2026 · ruled out
   Cap the batch Evidence

   Tried caps of 33, 25, 20, 15, 12, 10. The thread stayed below 0.4 at every cap. Capping drops tail candidates, but the threads that suppress it are the high-similarity on-topic ones — exactly the ones a cap keeps.

4. Jun 2026 · ruled out
   Score each thread in isolation Evidence

   Recovers the oblique thread (0.72) but floods the surface: unrelated threads also score 0.57–0.72 on their own and clear the floor. Removing the batch removes the discrimination too.

5. Jun 2026 · partial
   Prompt variants Evidence

   Reasoning-first, tension-crediting, and “movement” framings got the judge’s prose right — for the oblique thread it wrote a precise account of the counter-pressure. But the number stayed unreliable (mean 0.38 across runs, high variance). The reasoning was sound; the magnitude was the weak link.

6. Jun 2026 · the pivot
   Classify the kind, map to a fixed score Evidence

   Replace the freehand number with a fixed vocabulary of relationship kinds, each mapped to a fixed score. A classification question doesn’t depend on the other threads in the call, so it survives batching. Two kinds land below the floor, three above it, so nothing downstream changes.

7. Jun 2026 · verification
   Three-run eval, then ship behind a flag Evidence

   An offline eval ran the production judge over real recent artefacts, three runs each. The scheme read each relationship per-artefact rather than blanket-generously, with high label stability. Shipped code-only behind a feature flag, now on in production.

Hypotheses and experiments

The decision

The judge now classifies each thread’s relationship to the artefact from a fixed vocabulary, and a lookup maps the kind to the same score column the surface already used:

dead_end             0.00   drop    no shared stake; incidental vocabulary
echo                 0.20   drop    only restates what the thread already holds
direct_development   0.60   keep    pushes the thread’s own question forward
complication         0.85   keep    tensions / complicates / opposes the thesis
cross_domain_ground  0.90   keep    a mechanism or frame from another domain

The two non-resonant kinds map below the 0.4 floor and the three good-neighbour kinds above it, so the score column, the floor, the write/rollback path and the surface are all untouched — the verdict’s shape changed, not the plumbing. The prompt tells the judge explicitly that a complication or a cross-domain grounding is a prime neighbour, not a weak one, and to judge each thread on its own rather than rank them against each other.

A second, smaller change rode along: the surfaced reasons were long and led with a restatement of the thread (“This thread explores X…”). The prompt now caps each clause to a short span and tells the judge to state the move, not the topic. A response-wide token cap can’t do this — it would truncate the tail rather than tighten each reason — so the lever is the prompt.

What the eval showed

The bellwether is that one oblique thread across the first two artefacts: the image judge keeps it (complication), the book judge drops it (dead_end), both stable and both defensible — the image genuinely complicates the thread’s thesis, while the book is adjacent but doesn’t engage its actual stake. The scheme reads the real relationship per artefact rather than being blanket-generous. Agreement with the old scalar judge is high wherever the scalar was confident; the divergences sit in the scalar’s 0.25–0.38 mush, where the categorical judge pulls up real groundings the float threshold had buried. Evidence

Evidence and implementation trail

Sources are generalised for public release — internal file names, identifiers, thread and artefact titles are omitted or described. Each item is referenced inline above by its label.

Outcome, open questions, revisit triggers

Outcome

The categorical judge is live in production behind its flag, as a drop-in replacement for the scalar judge — same score column, same floor, same surface. The production miss that started this is recovered. Evidence

Open questions

Keep density. Keep rates ran 88% / 82% / 52% across the test artefacts — they track how genuinely connected an artefact is. When many links are real, showing all of them at once is a surface problem, to be handled by ranking and pagination, not by the judge dropping good links. Two density levers were tried and rejected: raising the floor (hides the most-connected hub threads, cuts unevenly) and tightening the pre-judge similarity filter (strictly recall-negative, since the judge already drops dead ends). Evidence

Label stability. A few artefacts show non-unanimous modal labels across runs, all at category boundaries (complication ↔ cross-domain ground, development ↔ complication). It doesn’t change keep/drop — both sides keep — but it’s worth watching. Unresolved

The score values are knobs. The mapping (0.6 / 0.85 / 0.9 above the floor, 0.0 / 0.2 below) is tunable; the eval is the place to re-tune it against more libraries. Unresolved

Revisit triggers

• The density surface gets built — that’s where ordering and pagination should absorb high keep counts.
• The category boundaries prove unstable enough to affect what users see.
• A larger library set suggests the kind→score mapping needs re-tuning.

Method and limitations

This devlog was reconstructed by Claude from a maintained record of the work, version history, and a check against the current code. The judge was verified live behind its flag at the time of writing.

• Internal file names, symbol names, ticket identifiers, and the specific thread and artefact titles are generalised for public release. Model names are public products and are kept.
• The result tables are point-in-time eval outputs on a handful of artefacts, three runs each — illustrative of the effect, not a standing benchmark.
• The score-floor and similarity-filter findings come from single-artefact sweeps; they show direction, not a tuned optimum.