Semantic entity resolution at 0.85 cosine falsely merges distinct organizations

[DarrenZal]

Summary

knowledge_router.py:1252 accepts a semantic match at confidence >= 0.85:

if resolved_uri and confidence >= 0.85:

For institutional names — which share heavy lexical and semantic structure (Government of X, Ministry of Y, Canadian Z Association) — 0.85 on text-embedding-3-large is far too permissive. A bulk ingest on 2026-08-24 (279 people, ~1,000 organizations from victoriaforum.ca) produced 28 false merges attributable to that single run, out of 318 semantic resolutions in the window (~9%).

Confirmed false merges

requested name	bound to	confidence
Government of Singapore	government-of-british-columbia
Government of Alberta	government-of-british-columbia
Government of Nunavut	government-of-british-columbia
Ministry of Foreign Affairs	ministry-of-education
Ministry of Energy and Climate Solutions	ministry-of-education
Department of National Defence	department-of-fisheries-and-oceans
Canadian Bar Association	canadian-standards-association
National Bank of Canada	national-research-council-canada
University of Texas at Austin	university-of-illinois-at-urbana-champaign
Order of St. John	order-of-ontario
Global Centre for Pluralism	global-centre-of-indigenomics	0.87
David Hughes (Person)	david-hutchens	0.94
John Richardson (Person)	john-rich	0.92

Elsewhere in the same log window (other ingests): Poland → portland, China → chinatown, Artin → martin, Groupoid → group.

Why it matters

This is silent corruption of pre-existing entities, not just a new-entity naming annoyance.

David Hutchens is a real, distinct person in the graph — author of Circle of the 9 Muses, with his own facts. The 0.94 match dumped 17 facts belonging to David Hughes (President & CEO of Generate Canada, a Victoria Forum speaker) onto him. Nothing surfaced this: the write succeeded, the log line reads like a successful resolution, and both people now read as one.

Same shape for Global Centre for Pluralism → Global Centre of Indigenomics: two unrelated organizations, one of which already carried a real affiliation fact.

Reproduce

grep "multi-tier resolved" <koi stderr log> \
  | grep -oP "resolved '\K[^']+|entity:\K[a-z0-9-]+"

Flag any resolution where the target URI slug shares no distinctive token with the requested name (after stripping stopwords like the/of/university/institute/canadian/government).

Suggested directions

1. Raise the floor for Organization/Location. These names are compositional; embedding similarity is a poor discriminator. 0.85 is defensible for freeform Concepts, not for institutions.
2. Gate semantic acceptance behind a lexical check. Require at least one shared distinctive token (or a Jaro-Winkler floor) between requested name and matched name before accepting a tier-2 match. This alone kills every row in the table above.
3. Treat a numeric/proper-noun mismatch as disqualifying — Alberta vs British Columbia, Singapore vs British Columbia, Texas vs Illinois are distinguishing tokens, not noise.
4. Log louder. skipped create_new currently reads as success. A resolution that overrode an explicit entity_type hint or matched below a stricter secondary check deserves a WARNING and ideally a review queue.

POST /register-entity with exact_only: true is an effective workaround for callers who know the name is canonical, and was used to repair all 31 bad merges from this run.

Context

Found while indexing victoriaforum.ca into personal-koi. Repair path used: pre-create with exact_only, retract the mis-pointed facts, rewrite so they resolve tier-1 exact.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Mo9wkRCWcZJ33k7QWZw5uz

[DarrenZal]

Root cause found — it is not the 0.85 threshold, it is a 29-word vocabulary

Correcting my own report above. The guard that should catch these already exists and already runs: resolve_entity_multi_tier applies passes_semantic_match_guard_with_policy(..., passes_token_overlap_strict) at the Tier-2b acceptance point (api/resolution_primitives.py:704), and it composes a person-name guard, a token-overlap policy, passes_distinctive_token_check, and a short-name margin. The logic is sound.

What fails is its vocabulary. GENERIC_NAME_TOKENS (api/resolution_primitives.py:234) has 29 entries:

a alliance and association camp center centre coalition collective college
council for foundation fund group in initiative institute lab labs network
of program project society team the trust university

passes_distinctive_token_check rejects only when the distinctive sets are disjoint. So:

pair	distinctive sets today	disjoint?
Government of Alberta vs Government of British Columbia	{government, alberta} vs {government, british, columbia}	no — shares government → accepted
Ministry of Foreign Affairs vs Ministry of Education	{ministry, foreign, affairs} vs {ministry, education}	no → accepted
Department of National Defence vs Department of Fisheries and Oceans	shares department	no → accepted

Missing from the list, and each one is load-bearing for a confirmed false merge:

government ministry department academy office bureau order national
canadian canada federal provincial commission agency authority board
committee bank service(s) research studies affairs public

Add government and the Alberta/BC pair becomes {alberta} vs {british, columbia} — disjoint — rejected. That is a data change to a set literal, not a logic change, which is a much smaller blast radius than the threshold adjustment I suggested in the issue body.

Why I am not attaching a patch

I tried to validate the extension by replaying the 32 confirmed pairs through passes_semantic_match_guard_with_policy directly, and the harness reported that it would reject Adam Olsen → Adam Olson — a merge that actually occurred in production at confidence 0.96 and was correct (it unified a misspelled single-fact stub with the real person).

The cause is that I supplied synthetic similarity values (0.90–0.94) where the real ones ranged 0.87–0.96, and the short-name branch keys off exactly that margin (similarity < threshold + 0.03). So the before/after counts my harness produced are an artifact of the inputs I invented, not evidence about the change.

Hand-simulation is the wrong instrument here. This repo already has the right one: the resolver_shadow machinery (currently enabled: true, sample_rate: 0.1), which is how the 2026-08-23 strict-vs-legacy semantic cutover was justified — "a semantic replay of 204 observations found 17 divergences, ALL of them Meeting" (evidence/resolver-shadow/semantic-20260823.log).

Recommendation: land the GENERIC_NAME_TOKENS extension behind the same shadow evaluation — measure legacy-vs-extended divergence on real traffic with real similarities, confirm the divergences are the institutional false merges and not legitimate variants, then flip. The 31 repaired merges from the 2026-08-24 Victoria Forum ingest make a good labelled evaluation set.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Mo9wkRCWcZJ33k7QWZw5uz

[DarrenZal]

Correcting again: it is the FUZZY tier, and no threshold separates the classes

Two things in my earlier comments were wrong. Replaying the live registry settled it.

1. Wrong tier

The false merges do not come through the semantic tier. mode=semantic in the
knowledge-add multi-tier resolved … log line only records that an embed function was
available, not which tier matched. The per-type thresholds are:

type	fuzzy (similarity_threshold)	semantic
Organization	0.85	0.92
Person	0.92	0.92
Location	0.90	0.92
Concept	0.75	0.88

Observed confidences on the bad merges cluster at 0.87–0.94 — mostly below the 0.92
semantic threshold. They are Jaro-Winkler fuzzy matches, admitted because
Organization's fuzzy floor is 0.85, seven points below Person's.

A semantic-tier replay of the whole registry produces zero observations for the
known-bad names, which is the proof: scripts/replay_resolver_shadow.py --tier semantic
never sees them.

2. Raising the threshold does not work

A full fuzzy replay (--tier fuzzy, whole Organization corpus) surfaces 80 distinct
pairs the current policy would merge. Hand-scoring them:

• merges that conflate different organizations: JW 0.853 – 0.993
• merges that are correct: JW 0.354 – 1.000

The distributions overlap across almost the entire range. Raising Organization to 0.92
would kill 22 of 24 bad merges and break roughly 25 good ones. There is no cutoff.

The reason is visible in the data — this graph is domain-clustered, so the bad pairs
share real distinctive tokens that passes_distinctive_token_check cannot strip:

salt spring island farmland trust  ->  salt spring ai          (shares: salt spring)
first nations major projects coal. ->  first nations finance…  (shares: first nations)
salish sea hub                     ->  salish sea life         (shares: salish sea)
coast salish gathering             ->  coast salish peoples    (shares: coast salish)
regen network / regen foundation   ->  regen                   (both collapse)

Extending GENERIC_NAME_TOKENS (my previous suggestion) still helps the
government/ministry/department family, but it cannot touch these — the shared tokens
are meaningful words that happen to be common in this corpus.

Also surfaced: at least one cross-type leak, opencivics labs (Organization)
resolving to a person- URI.

What I am not doing

I built two candidate guards and scored both — and both scoring runs were wrong,
because I was labelling the pairs myself. The second harness charged a rule with
"breaking" three merges that are in the confirmed-bad set, because I had only labelled
one direction of each pair.

Whether Regen Foundation and Regen Network are one entity is not something the
resolver author can decide from the strings. So the next step is a labelled set,
not another rule: all 80 pairs are written up for operator adjudication, 32 pre-filled
from verified evidence, 48 open. Once labelled, candidate guards score mechanically
and the winner ships against real data instead of my guesses.

Reproduce:

PYTHONPATH=. venv/bin/python scripts/replay_resolver_shadow.py \
    --sample 6000 --tier fuzzy --entity-type Organization --out /tmp/fuzzy_org.log

🤖 Generated with Claude Code

https://claude.ai/code/session_01Mo9wkRCWcZJ33k7QWZw5uz