W-559: merge Emoogle concept keywords into the emoji corpus

[wr]

Closes #182.

Searching by intent instead of by name mostly failed. Emojibase's tags come from CLDR and are deliberately literal, so :deploy, :ghosting, and :urgent returned nothing useful. This merges Emoogle's keyword set (MIT) into the same t array, adding 7,868 keywords net of what we already had.

What it fixes

Query	Before	After
deploy	—	🚀 🚢
ghosting	—	👻
urgent	—	🚨
launch	—	🚀

Normalization

Two things had to be handled in build_emoji_db.py:

• Emoji keys. Emoogle keys on the character, not the hexcode, and disagrees with emojibase about the FE0F variation selector (⚓ vs ⚓️). Both sides get it stripped before matching — that takes the match rate to 100% of Emoogle's 1,872 entries.
• Untypable phrases. Space isn't a name char, so :ship it: can't be captured. Phrases fold to underscore forms (to_the_moon), punctuation becomes a separator, and anything over three words is dropped — which removes Emoogle's 261 flag: … country names that our shortcodes already cover.

Emojibase's own tags are deliberately not run through the same fold: it would eat the keycap digit tags 0–9 and thumbsdown's -1.

Cost

	haystacks	per-keystroke
before	15,608	0.881 ms
after	23,947	1.279 ms

Measured with an -O build of FzyScorer over the real corpus, 20 representative queries, median of 5 runs on an M-series Mac, both figures taken in the same session (absolute numbers drift with machine load). Bundle grows 48 KB. The upstream file is SHA-pinned like every other source and was scanned for unsafe unicode before pinning (clean — ASCII plus a few accented letters and curly quotes).

Test plan

• scripts/run-tests.sh — 277 tests pass, 3 new
• EmojiDatabaseTests.corpusCarriesSemanticKeywords / .mergedKeywordsAreTypable guard the merge and the no-spaces invariant
• FuzzyMatcherTests.emoogleConceptKeywordSurfacesEmoji asserts the concepts actually land in the top 12 under real ranking, not just that the tag exists
• Rebuild is deterministic — running build_emoji_db.py twice yields an identical file hash

Not in this PR

Still open from the issue, tracked separately: query-time stemming (:ghosted: can't reach a ghost keyword, since fzy rejects a needle longer than its haystack), and slang (lfg, cozy, yeet, vibe) which no open dataset covers cleanly and needs hand-curation.

Refs W-559

---

Second commit: a relevance floor

Adding keywords exposed an older problem. With ~23k haystacks, something always matches as a scattered subsequence, so a query with no real answer still fills the picker:

query	before	after
:yeet:	🐞 💛 🪖 🥽 🫢 ❣️ 🤭	(nothing)
:lfg:	🥬 🤛 🍃 🇬🇫 📭️ 📪️ 🏌️	(nothing)
:cursed:	📈	(nothing)

🐞 is lady beetle with y‑e‑e‑t threaded through it. An empty picker is the honest answer — the user keeps typing instead of reading junk.

rankedResults now drops any candidate scoring below relevanceFloor × needle.count.

Picking the threshold

The two populations overlap: a one-character typo costs about what a junk subsequence scores. So this is a tolerance dial, not a clean separator. Swept it against 12 typo queries and 9 no-answer queries:

floor	typos kept	junk rows shown
none	10/12	50
0.60	10/12	10
0.65	9/12	10
0.72	9/12	4
0.85	3/12	4

0.60 keeps every typo that was reachable at all (roket → 🚀, sml → 😄, thnk → 🤔) and clears ~90% of the noise. Tightening further starts eating typos, which users hit far more often than they hit no-answer queries. 4 of the 10 rows remaining at 0.60 are :zzz: → 💤😴🥱🛌, which is a correct result I'd initially mislabeled as junk.

Also swept 65 common queries (smile, heart, deploy, rocket, …) comparing top-12 before and after: zero rows change. The floor is invisible on queries that have a real answer.

Details

• Aliases bypass the floor — an explicit alias is stated intent, and its +6.0 bonus would make the comparison meaningless.
• Checked before the frequency boost, so a heavily-used emoji can't drag an unrelated match back above the line.
• Off below 3 characters, where scores are low by construction and the prefix tier carries ranking.
• No measurable perf change (one comparison per candidate).

Test plan

• 280 tests pass (277 + 3 new)
• queryWithNoRealMatchReturnsNothing — yeet / lfg / cursed return empty
• floorKeepsTypoTolerance — roket → 🚀, sml → 😄, thnk → 🤔 survive
• floorSpares2CharQueries — the short-needle exemption holds

---

Third commit: review fixes

Codex reviewed the branch and found two real defects. Both fixed, both now covered by tests.

Aliases bypassed the floor. The floor exempted any match whose winning haystack was a custom alias, on the theory that an alias is stated intent. But the exemption applied to fuzzy alias matches too — cek scores 0.835 against an alias named checkered_flag (floor 1.80), and the +6.0 alias bonus then carried it to the top. So defining any alias reopened exactly the junk-match hole this PR closes.

The floor is now measured on the unbonused fzy score. The alias bonus lifts ranking, not relevance; an alias term still has to resemble the query. AliasTests.aliasDoesNotBypassTheRelevanceFloor covers it.

Dedup dropped 471 typable keywords. emoogle_keywords normalized both sides before comparing, but emojibase tags ship unnormalized (deliberately — normalizing them eats the keycap digit tags 0–9 and -1). So a raw untypable tag suppressed the only typable spelling of the same concept:

emoji	emojibase tag (ships as-is)	Emoogle form (was suppressed)
🕛	12:00 — a colon ends capture	12_00
🕛	twelve o'clock — space ends capture	twelve_o'clock

Neither spelling was reachable from a :query:. Dedup now compares against the raw spellings that actually ship, so the typable variant survives alongside. Merged keywords 7,868 → 8,339. EmojiDatabaseTests.untypableTagDoesNotSuppressItsTypableForm covers it.

282 tests pass.

[linear-code]

W-559

[wr-claude-reviewer]

Clean implementation. The Emoogle merge is well-scoped: SHA-pinned upstream, correct FE0F normalization on both sides, multi-word keyword folding that matches what the trigger can actually capture, deduplication that covers shortcodes + label + existing tags, and three layers of test coverage (corpus presence, no-space invariant, end-to-end search). The deliberate asymmetry — normalizing Emoogle keywords but leaving emojibase's raw tags alone — is the right call and correctly documented in CLAUDE.md.

A few nits:

mergedKeywordsAreTypable space-check is scoped to one emoji. The !rocket.tags.contains { $0.contains(" ") } assertion only protects rocket (1F680). A broader guard would also cover Emoogle keywords added to other emoji, and would catch any future emojibase tag that introduces a space:

#expect(EmojiDatabase.shared.all.allSatisfy { emoji in
    emoji.tags.allSatisfy { !$0.contains(" ") }
})

(Note: emojibase's own tags are not normalized today, so a multi-word emojibase tag would produce an unreachable haystack rather than a correctness bug — still worth catching.)

Implicit contract between KEYWORD_SEPARATORS and KeyMonitor.isNameChar. The comment notes they must stay in sync, but nothing enforces it. If isNameChar ever gains a new allowed character (e.g. .), Emoogle keywords containing that character would be silently dropped or split rather than preserved. Low-risk today, but worth a note in a future cross-reference.

[wr-claude-reviewer]

Round 2 follow-up — previous findings still unresolved (both nits), no new blocking issues.

New commit (de9c955 — relevance floor)

The floor implementation is correct. Key design decisions verified:

• Linear scaling (relevanceFloor * Double(needle.count)) is appropriate — fzy scores scale with needle length, so a per-character floor is the right comparison unit.
• Aliases bypass the floor — correct. An alias is declared intent; the +6.0 bonus would also render the comparison meaningless.
• Floor applied before frequency boost — correct. A popular-but-unrelated emoji can't be rescued by usage history.
• Disabled below 3 characters via floor = -Double.infinity so baseScore < floor is never true. The 2-char exemption is tested.
• prefixBestIsAlias tracking is wired up correctly alongside prefixBestScore/prefixBestDisplay; the alias bypass propagates through both the prefix and non-prefix paths.

Tests cover the three critical properties: junk queries return nothing, one-character typos survive, short needles are exempt. The sweep against 65 common queries (zero rows change) is a strong signal that the floor is invisible on real input.

Prior nits — still unresolved

These were flagged in round 1 and remain unchanged:

mergedKeywordsAreTypable scope (EmojiDatabaseTests.swift:84). The space-containment check is still scoped to rocket only. A full-corpus guard would catch a future emojibase tag or Emoogle regression on any emoji:

#expect(EmojiDatabase.shared.all.allSatisfy { emoji in
    emoji.tags.allSatisfy { !$0.contains(" ") }
})

(Note: emojibase's pre-existing tags skip normalize_keyword, so a multi-word emojibase tag would produce an unreachable haystack rather than a crash — still worth catching early.)

KEYWORD_SEPARATORS/KeyMonitor.isNameChar implicit contract (build_emoji_db.py:157). Still no cross-reference. If isNameChar gains a new allowed character, keywords containing it would be silently split or dropped rather than preserved. Low risk today.

[wr-claude-reviewer]

Round 3 follow-up — previous findings partially addressed (one nit resolved, one still open). No new blocking issues.

Third commit (c8aead0 — review fixes)

Both defects from the prior round are correctly fixed.

Alias bypass fix (FuzzyMatcher.swift): unbonusedScore carries the raw base fzy score (pre-alias-bonus). The floor comparison at if unbonusedScore < floor { continue } is measured against that raw score, so the +6.0 alias bonus lifts ranking but can no longer rescue a junk subsequence match that wouldn't otherwise clear the floor. The tracking is wired through both the prefix and non-prefix paths (prefixBestUnbonused / bestUnbonused). aliasDoesNotBypassTheRelevanceFloor covers the failure mode.

Dedup fix (build_emoji_db.py): emoogle_keywords now builds have from the raw existing spellings ({term.lower() for term in existing}), so an untypable emojibase tag like 12:00 no longer suppresses Emoogle's typable 12_00 — they're different strings when compared raw. untypableTagDoesNotSuppressItsTypableForm covers it.

Both fixes are minimal and targeted. The docstring on emoogle_keywords correctly explains the deliberate asymmetry.

Prior nits status

KEYWORD_SEPARATORS/isNameChar cross-reference (build_emoji_db.py) — ✅ resolved. The comment # Mirrors \KeyMonitor.isNameChar` — the only characters a `:query:` can contain.` is now present inline.

mergedKeywordsAreTypable scope (EmojiDatabaseTests.swift:87) — still open (third consecutive round). The space-containment check is still scoped to rocket only. This is a nit, not a blocker — a multi-word emojibase tag would produce an unreachable haystack rather than a crash — but widening it to the full corpus is a one-liner that makes future regressions self-documenting:

#expect(EmojiDatabase.shared.all.allSatisfy { emoji in
    emoji.tags.allSatisfy { !$0.contains(" ") }
})

[gabrieleconcli]

love this one!

[gabrieleconcli]

TIL ^