Repository navigation
feat(find): index column names + semantic_type in dataset search - #95
Merged
Merged
Conversation
The topic_index already carries full column defs (~6.2k across the Lab)
with name, role, semantic_type, description — but find() never scored
them, so a query matching only a column name/type returned nothing.
Columns are now a scored field on datasets (col name+desc as text,
semantic_type as tag), unlocking queries like:
find("cig") → anac_aggiudicatari (cig_code column)
find("codice_fiscale") → anac_aggiudicatari (fiscal_code column)
No new pipeline, no artifact growth — pure query-time change on data
already present. Level A only; distinct-value extraction (level C) is
explicitly out of scope, that's toolkit_query's job.
Profiling showed 95% of search_index time in matching, dominated by _deaccent (unicodedata.normalize + combining per char) re-running on every field of every entity for every query — even pure-ASCII fields. - _deaccent: skip normalize when text.isascii() (~99% of Lab fields) - _field_words: optional per-search word_cache keyed by field value; search_index threads one shared dict so identical values (source_id, category, registry_source repeated across datasets) tokenize once - _score_entity: precompute token forms once, tokenize each field once per entity (was once per token×field); drop the double _field_words call in the prepared listcomp Wall-clock on real CI artifact (436 datasets, 10-entity limit): ~150ms → ~70ms per query. _field_words calls 173k → 97k (cache hits are dict lookups), _deaccent 173k → 63k (rest are ascii no-ops). Behavior unchanged — same scores, same hits on all regression queries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sintesi
find()non scorreva le colonne del dataset, nonostante iltopic_indexporti già ~6.200 definizioni di colonna conname/role/semantic_type/description. Una query che matchava solo un nome colonna o un semantic_type restituiva 0 risultati.Le colonne ora sono un campo scorato: nomi+descrizioni come testo di scoperta,
semantic_typecome tag strutturato. Query comefind("cig")ofind("codice_fiscale")ora trovanoanac_aggiudicatari(colonnecig→cig_code,codice_fiscale→fiscal_code) che prima uscivano solo se quei termini comparivano in slug/name/description/tags.Contesto collegato
Livello A della valutazione complessità/ROI ricerca. Livello B (arricchire
semantic_typevuoti nei cataloghi) e C (estrazione valori distinti) fuori scope — C in particolare duplicherebbe il ruolo ditoolkit_query.Cosa cambia
navigation.py:search_indexDettaglio
_columns_index_text(columns)→(names+descs, semantic_types)search_index: merge delle entry full aggiungecolumns; scoring dataset aggiunge(col_text, "text")e(col_sem, "tag")by_slugintoccato (le colonne restano solo nelle entry full, che sono la fonte primaria)Impatto su artifact upstream
Verifica
Smoke su artifact CI reale
ciganac_aggiudicatari(colonnacig)anac_aggiudicatariscore 35codice_fiscaleanac_aggiudicatarianac_aggiudicatariscore 35appalti pubblici ANAC/bandi di gara/ negativiRank 35 (text) vs 80 (slug) per i match su colonna: corretto, un match su schema vale meno di un match su slug.
Checklist PR
Note per chi revisiona
columnssuanac_bandi_garaper coprire nome colonna (importo) e semantic_type (cig); le entry senza colonne degradano graceful a 0 nel campo.toolkit_query(SQL), non find — non si duplica qui.