W-559: merge Emoogle concept keywords into the emoji corpus

CLAUDE.md

@@ -133,7 +133,13 @@ TextInserter (synthetic CGEvents) — or one of the easter-egg effects under Sou
 
 ### Emoji search
 
-`EmojiDatabase.indexed` is an array of `IndexedEmoji`, each carrying a real `Emoji` plus precomputed `EmojiHaystack` entries (lowercased `[Character]` arrays for every shortcode + label). This is built once at DB load. `FuzzyMatcher.search` iterates `database.indexed` and runs `FzyScorer.score(needle:haystack:)` — a Swift port of John Hawthorn's fzy scoring algorithm. Critically, the search loop **never allocates strings**, which keeps per-keystroke cost in the microseconds.
+`EmojiDatabase.indexed` is an array of `IndexedEmoji`, each carrying a real `Emoji` plus precomputed `EmojiHaystack` entries (lowercased `[Character]` arrays for every shortcode + label). This is built once at DB load. `FuzzyMatcher.search` iterates `database.indexed` and runs `FzyScorer.score(needle:haystack:)` — a Swift port of John Hawthorn's fzy scoring algorithm. Critically, the search loop **never allocates strings**.
+
+Cost scales with total haystack count (~24k for the English corpus: shortcodes + labels + tags). Measured ~1.3 ms per keystroke on an M-series Mac for a 2+ char query, single-threaded on the main thread — against ~0.9 ms for the 15.6k-haystack corpus before semantic keywords landed. Adding preferred locales or another keyword source moves that number roughly linearly, so benchmark before growing the corpus again; absolute figures drift with machine load, so always measure old and new in the same session.
+
+`FuzzyMatcher.relevanceFloor` drops any match scoring below 0.60 per needle character. Without it a query with no real answer still fills the picker, because across ~24k haystacks something always matches as a scattered subsequence (`:yeet:` → 🐞, via y‑e‑e‑t inside "lady beetle"). The value is a tolerance dial, not a clean separator — a one-character typo costs about what a junk subsequence scores, so it's set permissively enough to keep `roket` → 🚀. It's measured on the *unbonused* fzy score, so the +6.0 alias bonus can't smuggle a junk match past it, and checked before the frequency boost so usage can't either.
+
+Semantic keywords in `t` come from two upstreams, merged by `build_emoji_db.py`: emojibase's literal CLDR tags, and Emoogle's concept keywords (MIT) which cover associations CLDR omits (`deploy` on 🚀, `ghosting` on 👻). Emoogle keys are emoji characters with inconsistent FE0F, and its phrases contain spaces and punctuation that `:query:` can't capture — `emoji_key()` and `normalize_keyword()` handle both. Emojibase's own tags are deliberately **not** run through `normalize_keyword` (it would eat the keycap digit tags `0`–`9` and `-1`).
 
 Special "pinned" rows are appended after the fzy results when the lowercased query hashes to an entry in `EggIndex.prefix` (which already covers 3+ char prefixes of every registered easter-egg trigger). The pinned hexcode is the opaque id (`k01`, `k02`, …) returned by `EggIndex`, so `Engine.insert` and `PickerView` route on that id without ever string-matching the keyword. Symbols (★ ⌘ ⌥ etc.) live in `SymbolsDatabase.indexed()` and get prepended to the corpus only when `PrefsKey.symbolsEnabled` is true.
 

README.md

@@ -89,7 +89,7 @@ scripts/run-locale.sh fr   # or de, ja, ar, zh-Hans, etc.
 
 ## Credits
 
-emojibase, Sparkle, KeyboardShortcuts, GIPHY, and a Swift port of fzy.
+emojibase, [Emoogle](https://github.com/xitanggg/emoogle-emoji-search-engine), Sparkle, KeyboardShortcuts, GIPHY, and a Swift port of fzy.
 
 ## Donate
 

Sources/Mojito/EmojiDB/FuzzyMatcher.swift

@@ -79,6 +79,28 @@ struct FuzzyMatcher {
     /// heavily-used built-in when the query is the alias term.
     static let aliasBonus = 6.0
 
+    /// Minimum fzy score per needle character for a match to be shown at all.
+    ///
+    /// Across ~23k haystacks *something* always matches as a scattered
+    /// subsequence, so without a floor `:yeet:` returns 🐞 (y‑e‑e‑t threaded
+    /// through "lady beetle") and `:lfg:` returns 🥬. An empty picker is the
+    /// honest answer there — the user keeps typing instead of reading junk.
+    ///
+    /// The value is a tolerance dial, not a clean separator: the two
+    /// populations overlap, because a one-character typo costs roughly what a
+    /// junk subsequence scores. Measured over the corpus, `roket` → 🚀 lands
+    /// at 0.78 and `sml` → 😄 at 0.62, while junk mostly sits below 0.60.
+    /// Anything above ~0.65 starts eating typos, which are far more common
+    /// than the junk-only queries the floor exists to catch — hence a
+    /// deliberately permissive cut that clears ~90% of the noise and keeps
+    /// every typo that was reachable at all.
+    private static let relevanceFloor = 0.60
+
+    /// Below this the floor is off. Short needles score low by construction
+    /// (fewer characters to earn consecutive bonuses) and are driven by the
+    /// prefix tier anyway.
+    private static let floorMinNeedle = 3
+
     private struct PinnedRow {
         let hexcode: String
         let character: String
@@ -246,12 +268,18 @@ struct FuzzyMatcher {
         var results: [Candidate] = []
         results.reserveCapacity(64)
 
+        let floor = needle.count >= floorMinNeedle
+            ? relevanceFloor * Double(needle.count)
+            : -Double.infinity
+
         for indexed in pool {
             var bestScore: Double = -.infinity
             var bestDisplay: String?
             var bestIsTag = false
+            var bestUnbonused: Double = -.infinity
             var prefixBestScore: Double = -.infinity
             var prefixBestDisplay: String?
+            var prefixBestUnbonused: Double = -.infinity
             for haystack in indexed.haystacks {
                 if haystack.isTag && !scanTags { continue }
                 guard let base = FzyScorer.score(needle: needle, haystack: haystack.chars) else { continue }
@@ -268,11 +296,13 @@ struct FuzzyMatcher {
                     if raw > prefixBestScore {
                         prefixBestScore = raw
                         prefixBestDisplay = haystack.display
+                        prefixBestUnbonused = base
                     }
                 } else if raw > bestScore {
                     bestScore = raw
                     bestDisplay = haystack.display
                     bestIsTag = haystack.isTag
+                    bestUnbonused = base
                 }
             }
 
@@ -282,18 +312,28 @@ struct FuzzyMatcher {
             let display: String
             let baseScore: Double
             let matchedIsTag: Bool
+            let unbonusedScore: Double
             if let prefixBestDisplay {
                 display = prefixBestDisplay
                 baseScore = prefixBestScore
                 matchedIsTag = false
+                unbonusedScore = prefixBestUnbonused
             } else if let bestDisplay {
                 display = bestDisplay
                 baseScore = bestScore
                 matchedIsTag = bestIsTag
+                unbonusedScore = bestUnbonused
             } else {
                 continue
             }
 
+            // Measured on the raw fzy score: the alias bonus is a ranking lift,
+            // not a relevance override, so an alias term still has to actually
+            // resemble the query. Otherwise defining any alias would restore the
+            // junk subsequence matches the floor exists to remove. Checked
+            // before the frequency boost too, so usage can't rescue junk either.
+            if unbonusedScore < floor { continue }
+
             var finalScore = baseScore
             if useFrequencyBoost, let count = usage[indexed.emoji.hexcode], count > 0 {
                 // Cap at +5.0 (~ one extra strong-bonus match) so popular

Tests/MojitoTests/AliasTests.swift

@@ -181,4 +181,15 @@ struct AliasIndexTests {
         let ranked = rank("check", result, usage: ["E_FLAG": 100])
         #expect(ranked.first == "E_CHECK")
     }
+
+    @Test func aliasDoesNotBypassTheRelevanceFloor() {
+        // The +6.0 alias bonus lifts ranking, not relevance: "cek" threads
+        // through "checkered_flag" as a scattered subsequence and scores under
+        // the floor, so defining the alias must not resurface it. Otherwise any
+        // alias would reopen the junk-match hole the floor closes.
+        let result = build([CustomAlias(alias: "checkered_flag", hexcode: "E_CHECK")])
+        #expect(!rank("cek", result).contains("E_CHECK"))
+        // The alias term itself still resolves.
+        #expect(rank("checkered_flag", result).first == "E_CHECK")
+    }
 }

Tests/MojitoTests/EmojiDatabaseTests.swift

@@ -64,4 +64,37 @@ struct EmojiDatabaseTests {
         // Haystacks are pre-lowercased `[Character]` arrays — one per shortcode.
         #expect(entry.haystacks.contains { $0.chars == Array("smile") })
     }
+
+    /// Concept keywords merged from Emoogle by `build_emoji_db.py`. A rebuild
+    /// that silently drops the merge would still pass every other test here.
+    @Test(arguments: [
+        ("1F680", "deploy"),      // 🚀
+        ("1F680", "launch"),
+        ("1F47B", "ghosting"),    // 👻
+        ("1F6A8", "urgent"),      // 🚨
+    ])
+    func corpusCarriesSemanticKeywords(hexcode: String, keyword: String) throws {
+        let emoji = try #require(EmojiDatabase.shared.byHexcode[hexcode])
+        #expect(emoji.tags.contains(keyword))
+    }
+
+    /// Space isn't a name char, so a multi-word keyword is only reachable in its
+    /// underscore form. `build_emoji_db.py` folds Emoogle's phrases on the way
+    /// in; emojibase's own tags predate that and are left as-is.
+    @Test func mergedKeywordsAreTypable() throws {
+        let rocket = try #require(EmojiDatabase.shared.byHexcode["1F680"])
+        #expect(rocket.tags.contains("to_the_moon"))
+        #expect(!rocket.tags.contains { $0.contains(" ") })
+    }
+
+    /// An emojibase tag that ships in an untypable spelling must not suppress
+    /// Emoogle's typable one. 🕛 carries the raw CLDR tag `12:00` — a colon ends
+    /// capture, so it's unreachable — and Emoogle's `12_00` is the only form a
+    /// `:query:` can express. Deduping the two on their normalized key dropped
+    /// 471 keywords this way.
+    @Test func untypableTagDoesNotSuppressItsTypableForm() throws {
+        let twelve = try #require(EmojiDatabase.shared.byHexcode["1F55B"])
+        #expect(twelve.tags.contains("12_00"))
+        #expect(twelve.tags.contains("twelve_o'clock"))
+    }
 }

Tests/MojitoTests/FuzzyMatcherTests.swift

@@ -134,6 +134,17 @@ struct FuzzyMatcherTests {
         #expect(search("happy").contains { $0.emoji.hexcode == "1F600" })
     }
 
+    @Test(arguments: [
+        ("deploy", "1F680"),     // 🚀 — concept, not in any shortcode or CLDR tag
+        ("ghosting", "1F47B"),   // 👻
+        ("urgent", "1F6A8"),     // 🚨
+    ])
+    func emoogleConceptKeywordSurfacesEmoji(query: String, hexcode: String) {
+        // Emoogle's keyword merge is what makes these reachable at all — none
+        // of them appear in the emoji's shortcodes, label, or emojibase tags.
+        #expect(search(query, limit: 12).contains { $0.emoji.hexcode == hexcode })
+    }
+
     @Test func relevantTagMatchOutranksLooseSubsequence() throws {
         // ":happ" — 😀 matches the exact tag "happy"; ♿️ matches "happ" only as
         // a scattered subsequence of its "handicapped" shortcode (h‑a‑..‑p‑p).
@@ -145,6 +156,32 @@ struct FuzzyMatcherTests {
         #expect(happyIdx < wheelchairIdx)
     }
 
+    @Test(arguments: ["yeet", "lfg", "cursed"])
+    func queryWithNoRealMatchReturnsNothing(query: String) {
+        // Across ~23k haystacks something always matches as a scattered
+        // subsequence — 🐞 for "yeet", 🥬 for "lfg". None of these words is in
+        // the corpus, so an empty picker is the correct answer.
+        #expect(realResults(search(query)).isEmpty)
+    }
+
+    @Test(arguments: [
+        ("roket", "1F680"),   // 🚀 dropped 'c'
+        ("sml", "1F604"),     // 😄 dropped vowels
+        ("thnk", "1F914"),    // 🤔
+    ])
+    func floorKeepsTypoTolerance(query: String, hexcode: String) {
+        // The floor is set below the cost of a one-character typo. Tightening
+        // it to separate junk perfectly would break these, which users hit far
+        // more often than they hit junk-only queries.
+        #expect(search(query, limit: 12).contains { $0.emoji.hexcode == hexcode })
+    }
+
+    @Test func floorSpares2CharQueries() {
+        // Short needles score low by construction, so the floor is off below 3
+        // characters — the prefix tier carries them instead.
+        #expect(!search("wo").isEmpty)
+    }
+
     @Test func tagMatchLabelsWithPrimaryShortcode() throws {
         // A row that matched via the "happy" tag must label itself with the
         // emoji's own primary shortcode (😀 → grinning), not the shared tag

scripts/build_emoji_db.py

@@ -6,6 +6,12 @@
 shortcodes into a single JSON optimized for fuzzy lookup. Output goes to
 Resources/Emoji/emoji.json.
 
+Semantic keywords come from two places: emojibase's own `tags` (literal, CLDR-
+derived) and Emoogle's keyword set (MIT), which adds everyday-concept
+associations CLDR deliberately omits — `deploy` on 🚀, `ghosting` on 👻,
+`urgent` on 🚨. Both land in the same `t` array; the Swift side scores them as
+penalized non-typable haystacks.
+
 Run this whenever you want to refresh the dataset:
     python3 scripts/build_emoji_db.py
 
@@ -21,16 +27,21 @@
 import hashlib
 import json
 import os
+import re
 import sys
 import urllib.request
 
 BASE = "https://raw.githubusercontent.com/milesj/emojibase/master/packages/data"
 EN_REPO = f"{BASE}/en"
+EMOOGLE = "https://raw.githubusercontent.com/xitanggg/emoogle-emoji-search-engine/main/data"
 SOURCES = {
     "compact":   f"{EN_REPO}/compact.raw.json",
     "iamcal":    f"{EN_REPO}/shortcodes/iamcal.raw.json",
     "emojibase": f"{EN_REPO}/shortcodes/emojibase.raw.json",
     "github":    f"{EN_REPO}/shortcodes/github.raw.json",
+    # Keyed by emoji character rather than hexcode, and inconsistent about the
+    # FE0F variation selector — `emoogle_keywords()` normalizes both.
+    "emoogle":   f"{EMOOGLE}/emoogle-emoji-keywords.json",
 }
 
 # Locales with localized shortcode coverage. `cldr-native` preserves
@@ -60,6 +71,10 @@
     "iamcal":    "c8181b1dabee299b7991739dd634a943c36a67f278995e7b1e48dc7f69b7d073",
     "emojibase": "5ea367e3866688e733a990bb099c36ffdee43e08ba1c03d48001b6fecf746fbe",
     "github":    "279d7669438a0f810db53aa62a12dbd40285270ea0598222575e9540781e3dfb",
+    # Emoogle (MIT). Keyword strings only — reviewed for unsafe unicode; the
+    # set is plain ASCII plus a handful of accented letters and curly quotes,
+    # which `normalize_keyword` folds or drops.
+    "emoogle":   "7a13ba1537583b0fc29f270a80f713a28c34d76fe4492c5da85ccbb25cd040b9",
     # Locale shortcode digests — Unicode-3.0 licensed, all data ultimately
     # sourced from Unicode CLDR. To bump: run `--print-shas`, review the
     # cached files, paste new digests here.
@@ -132,6 +147,59 @@ def fetch(name: str, url: str) -> object:
     return json.loads(raw)
 
 
+VARIATION_SELECTOR_16 = "️"
+
+# Typographic characters Emoogle uses that have a plain-ASCII equivalent a user
+# can actually type. Anything else outside the name-char set is a separator.
+KEYWORD_TRANSLITERATIONS = str.maketrans({"’": "'", "‐": "-", "“": "", "”": ""})
+# Mirrors `KeyMonitor.isNameChar` — the only characters a `:query:` can contain.
+KEYWORD_SEPARATORS = re.compile(r"[^a-z0-9_+'-]+")
+# Beyond this a keyword is a sentence, not something anyone types between
+# colons. Cuts Emoogle's 261 `flag: …` country names and the long official
+# emoji names, which our shortcodes and label already cover.
+KEYWORD_MAX_WORDS = 3
+
+
+def emoji_key(character: str) -> str:
+    """Match key for cross-dataset emoji lookup. Emojibase and Emoogle disagree
+    on whether FE0F is present (⚓ vs ⚓️), so it's dropped from both sides."""
+    return character.replace(VARIATION_SELECTOR_16, "")
+
+
+def normalize_keyword(word: str) -> str:
+    """Fold a semantic keyword into a form the trigger can actually capture, or
+    "" if it can't be one. Space isn't a name char, so `:ship it:` is untypable —
+    multi-word keywords become `ship_it`, which fzy also rewards with the
+    after-underscore word bonus."""
+    folded = word.lower().translate(KEYWORD_TRANSLITERATIONS)
+    parts = [p for p in KEYWORD_SEPARATORS.split(folded) if p]
+    if not parts or len(parts) > KEYWORD_MAX_WORDS:
+        return ""
+    joined = "_".join(parts).strip("_-'")
+    return joined if len(joined) > 1 else ""
+
+
+def emoogle_keywords(source: dict, character: str, existing: list[str]) -> list[str]:
+    """Emoogle keywords for `character` that aren't already covered, in Emoogle's
+    own relevance order. `existing` is every term already searchable on this
+    emoji (shortcodes, label, emojibase tags) in raw form.
+
+    Dedup compares against the *raw* existing spellings, because that's what
+    actually ships — emojibase tags are emitted unnormalized (see
+    `normalize_keyword`'s caveat about keycap digits). Normalizing both sides
+    would let an untypable tag like `12:00` suppress Emoogle's typable `12_00`,
+    leaving the concept unreachable from a `:query:` in either form."""
+    have = {term.lower() for term in existing}
+    out: list[str] = []
+    for word in source.get(emoji_key(character), []):
+        key = normalize_keyword(word)
+        if not key or key in have:
+            continue
+        have.add(key)
+        out.append(key)
+    return out
+
+
 def normalize_shortcodes(value) -> list[str]:
     if value is None:
         return []
@@ -154,8 +222,10 @@ def main() -> int:
     iamcal = sources["iamcal"]
     emojibase = sources["emojibase"]
     github = sources["github"]
+    emoogle = {emoji_key(char): words for char, words in sources["emoogle"].items()}
 
     out = []
+    emoogle_added = 0
     for entry in compact:
         hexcode = entry["hexcode"]
 
@@ -194,12 +264,23 @@ def main() -> int:
             if loc_list:
                 loc_codes[locale] = loc_list
 
+        tags = list(entry.get("tags", []))
+        semantic = emoogle_keywords(
+            emoogle,
+            entry["unicode"],
+            existing=shortcodes + [entry["label"]] + tags,
+        )
+        emoogle_added += len(semantic)
+
         item = {
             "h": hexcode,
             "e": entry["unicode"],
             "n": entry["label"],
             "s": shortcodes,
-            "t": entry.get("tags", []),
+            # emojibase's literal keywords first, then Emoogle's concept
+            # keywords — order is cosmetic (the Swift scorer ranks by fzy
+            # score, not position) but keeps diffs readable.
+            "t": tags + semantic,
             "g": entry.get("group", -1),
             "o": entry.get("order", 0),
             # True if the emoji has skin-tone variants in emojibase. The Swift
@@ -222,6 +303,7 @@ def main() -> int:
 
     size = os.path.getsize(OUT)
     print(f"Wrote {len(out)} emoji to {OUT} ({size/1024:.1f} KB)")
+    print(f"  {emoogle_added} Emoogle keywords merged (net of duplicates)")
     return 0