From d2d54d1f62a0b2e90ba05b56203cc0477195fc7d Mon Sep 17 00:00:00 2001 From: SkyeAv Date: Thu, 15 Oct 2026 11:12:46 -0700 Subject: [PATCH] docs: replace arrow notation with plain prose for readability Rewrites the chained '->' arrows that read as AI-generated text across README.md and docs/ into natural sentences, comma lists, and 'to'/'then' phrasing. Python return-type annotations, the quick_map example f-string, and KGX '--[predicate]-->' edge notation are code and stay untouched. - QC stage lists (exact -> fuzzy -> abbreviation -> SapBERT) become comma-separated stage names in README.md, docs/installation.md, docs/tutorial.md, docs/cli.md, and docs/api/qc.md - Pipeline and dataflow arrows in docs/fullmap.md, docs/cli.md, docs/api/fullmap.md become prose or comma lists - Input/output example arrows in config docs and code comments become 'becomes'/'maps to'/'yields' - Bidirectional '<->' pair notation becomes plain 'X / Y' or 'X and Y' - Schema-migration 'v3->v4' wording and the '[CLI Reference -> build-fullmap]' link label reworded No behavior change; markdown only. --- README.md | 6 ++-- docs/api/fullmap.md | 6 ++-- docs/api/lib.md | 4 +-- docs/api/qc.md | 12 ++++---- docs/cli.md | 15 +++++----- docs/configuration/advanced-example.md | 16 +++++------ docs/configuration/table.md | 16 +++++------ docs/examples.md | 2 +- docs/fullmap.md | 39 +++++++++++++------------- docs/installation.md | 2 +- docs/tutorial.md | 4 +-- 11 files changed, 62 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index b4aa99f5..a70c9b45 100644 --- a/README.md +++ b/README.md @@ -127,8 +127,8 @@ See the [Tutorial](https://skyeav.github.io/Tablassert/tutorial/) for the full w - **Declarative YAML configuration**: define data transformations without writing code - **Built-in entity resolution**: map free text to genes, diseases, and chemicals with standard CURIEs, taxonomic filtering, and provenance, backed by an embedded redb database -- **Optional quality control**: a four-stage audit (exact -> fuzzy -> abbreviation -> SapBERT embeddings) flags - low-confidence mappings +- **Optional quality control**: a four-stage audit (exact match, fuzzy match, abbreviation + expansion, then SapBERT embeddings) flags low-confidence mappings - **KGX compliance**: emits NCATS Translator-compatible node/edge NDJSON with Biolink categories and predicates - **Performance & reproducibility**: lazy Polars pipelines and a deterministic UV-based @@ -148,7 +148,7 @@ provides the `tablassert` command. Other extras are opt-in: | `cli` | `tablassert` command and rich terminal progress | `pip install "tablassert[cli]"` | | `rt` | CPU-compatible Polars runtime | `pip install "tablassert[rt]"` | | `aria2` | bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `pip install "tablassert[aria2]"` | -| `qc` | four-stage QC audit (exact -> fuzzy -> abbreviation -> SapBERT embeddings) | `pip install "tablassert[qc]"` | +| `qc` | four-stage QC audit (exact match, fuzzy match, abbreviation expansion, and SapBERT embeddings) | `pip install "tablassert[qc]"` | | `log` | loguru-backed file/progress logging (rotation, enqueue) | `pip install "tablassert[log]"` | Reaching a feature whose extra is missing never produces a bare `ModuleNotFoundError`: the failure diff --git a/docs/api/fullmap.md b/docs/api/fullmap.md index f04edab4..d113ed18 100644 --- a/docs/api/fullmap.md +++ b/docs/api/fullmap.md @@ -177,7 +177,7 @@ from tablassert.biolink import Categories db = Path("/path/to/fullmap/data/fullmap.redb") lf = pl.LazyFrame({"gene": ["TP53", "BRCA1", "EGFR", "KRAS"]}) lf = level_one(lf, "gene") # canonical tokenize, stem, deduplicate, and sort -lf = level_two(lf, "gene") # remove non-word chars -> "gene_two" column +lf = level_two(lf, "gene") # strips non-word chars into the "gene_two" column result = resolve(lf=lf, col="gene", db=db, taxon="9606", prioritize=[Categories.GENE], log=False).collect() @@ -198,7 +198,7 @@ print(result.select(["gene", "gene_name", "gene_category"])) Schema-v5 fullmap databases are rejected by the current resolver. Rebuild existing fullmaps to produce schema v6 before using them with this level-one contract. **`level_two` output** (column: `col + "_two"`): -- All non-word characters removed (`\W+` -> `""`) from the `level_one` result +- All non-word characters (`\W+`) removed from the `level_one` result - Used as fallback when `level_one` produces no match - Preferred for disease names and free text @@ -218,7 +218,7 @@ See [Quality Control](qc.md) for details. Single-shot inspection core behind the `tablassert quick-map` CLI command: resolve raw terms against a fullmap exactly as a build would, one result frame per input term. It runs the same op -chain a build runs per node column (normalization -> probe keys -> one batched redb fetch -> +chain a build runs per node column (normalization, probe keys, one batched redb fetch, then filter/rank/dedup), so the returned rows are the rows `build-kg` would emit for a cell holding that term under a `NodeEncoding` with the same settings. The whole input is ONE batched lookup, never a round trip per term. diff --git a/docs/api/lib.md b/docs/api/lib.md index 9c24b8c1..68b1bef4 100644 --- a/docs/api/lib.md +++ b/docs/api/lib.md @@ -110,8 +110,8 @@ result: list[dict[str, Any]] = resolve_many( prioritize=[Categories.GENE], ) -# result[0] -> {"original_gene": "TP53", "gene": "HGNC:11998", "gene_name": "TP53", ...} -# result[1] -> {"original_gene": "BRCA1", "gene": "HGNC:1100", "gene_name": "BRCA1", ...} +# result[0]: {"original_gene": "TP53", "gene": "HGNC:11998", "gene_name": "TP53", ...} +# result[1]: {"original_gene": "BRCA1", "gene": "HGNC:1100", "gene_name": "BRCA1", ...} ``` #### Consuming Results diff --git a/docs/api/qc.md b/docs/api/qc.md index 670e8eb7..25d1ac87 100644 --- a/docs/api/qc.md +++ b/docs/api/qc.md @@ -116,7 +116,7 @@ or fuzz.partial_token_sort_ratio(original, preferred) >= 80 _is_abbrev(original, preferred_name) or _is_abbrev(preferred_name, original) ``` -The matcher scans the short form right-to-left against the long form (case-insensitively); the first character of the short form must land on a word boundary of the long form. This rescues the class both fuzzy matching and embedding similarity can miss: `AML` <-> `acute myeloid leukemia`. +The matcher scans the short form right-to-left against the long form (case-insensitively); the first character of the short form must land on a word boundary of the long form. This rescues the class both fuzzy matching and embedding similarity can miss, the `AML` / `acute myeloid leukemia` pair. **Performance:** O(n) character scans per row, no model inference. @@ -181,10 +181,10 @@ validated = fullmap_audit( ``` Input: 1000 rows with entity mappings -Stage 1 (Exact): 700 pass -> 300 pending -Stage 2 (Fuzzy): 250 pass -> 50 pending -Stage 3 (Abbreviation): 10 pass -> 40 pending -Stage 4 (SapBERT): 30 pass -> 10 rejected +Stage 1 (Exact): 700 pass, 300 pending +Stage 2 (Fuzzy): 250 pass, 50 pending +Stage 3 (Abbreviation): 10 pass, 40 pending +Stage 4 (SapBERT): 30 pass, 10 rejected Output: 990 rows (700 + 250 + 10 + 30) ``` @@ -195,7 +195,7 @@ Output: 990 rows (700 + 250 + 10 + 30) |-------|--------|-----------|----------| | 1 | Exact match / rule-based | Highest | Standardized IDs, acronyms, CURIE-like inputs | | 2 | Fuzzy | Medium | Typos, word reordering | -| 3 | Abbreviation expansion | High | Abbreviation <-> full-name pairs | +| 3 | Abbreviation expansion | High | Abbreviation / full-name pairs | | 4 | SapBERT | High | Synonyms, paraphrases | ### Rejection Logging diff --git a/docs/cli.md b/docs/cli.md index 390f3339..bd7b5896 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -70,8 +70,9 @@ Every one of those invocations produces a database filtered by the checked-in By default `build-fullmap` looks for a prebuilt `fullmap.tar.zst` at `https://stars.renci.org/var/babel_outputs//fullmap//` (the version directory is the **installed Tablassert package version**, never hardcoded), verifies it against the -published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension, streaming zstd -> -tar with the GIL released (the decompressed tar never touches disk), then validating the extracted +published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension, streaming the +archive through zstd decompression and untarring it with the GIL released (the decompressed tar never +touches disk), then validating the extracted primary + shards against the force-build contract (exact `v6` schema, a recorded `build_id`, the exact shard set, per-shard `build_id` equality, and a `META.taxon_allowlist` identity matching the built-in allowlist) before atomically renaming them into place. If no prebuilt exists for this version (or the @@ -103,8 +104,8 @@ See [Fullmap](fullmap.md) for the data pipeline, output schema, and graph-config Use this to see what fullmap entity resolution will do with one or more terms (the rows a `build-kg` run would emit for a cell holding that term under the filters shown. Each term runs -through the exact build path (level-one/level-two normalization -> probe-key extraction -> redb -fetch -> filter, rank, dedup), one `rich` table per term in input order, with the term's normalized +through the exact build path (level-one/level-two normalization, probe-key extraction, one redb +fetch, then filter, rank, and dedup), one `rich` table per term in input order, with the term's normalized probe keys in the title so a miss is diagnosable. The whole input is one batched redb round trip, never one per term. A term with no matches prints a `no matches` line and still exits 0: a miss is a finding, not a failure; only usage errors and an unreadable fullmap exit 2. @@ -165,7 +166,7 @@ The positional `GRAPH-CONFIGURATION-FILE` (also `--configuration-file`, `-f`) is | --- | --- | --- | --- | --- | | `GRAPH-CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | n/a | Graph YAML | | `--release`, `-r` | Flag | No | `False` | Emit a slim, significant-only graph (drops `not_significant` edges before resolution) | -| `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact -> fuzzy -> abbreviation -> SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON: no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), no stray whitespace (verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies), and, when any section declares a `category_override`, no edge demoted to bare `biolink:Association` (a row escaped its pin and shipped without the class-specific slots `prune_to_class` nulled into `has_supporting_studies`; the assertion is graph-wide, so one pinned section also gates edges from unpinned sections) and fails the build (non-zero exit) on any violation | +| `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact match, fuzzy match, abbreviation expansion, then SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON: no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), no stray whitespace (verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies), and, when any section declares a `category_override`, no edge demoted to bare `biolink:Association` (a row escaped its pin and shipped without the class-specific slots `prune_to_class` nulled into `has_supporting_studies`; the assertion is graph-wide, so one pinned section also gates edges from unpinned sections) and fails the build (non-zero exit) on any violation | | `--log`, `-l` | Flag | No | `False` | Enable verbose per-section logging; requires the `[log]` extra, and without it Tablassert produces no logs | | `--head`, `-hd` | Flag | No | `False` | Fast output-shape preview: <=5 random rows/section, cached to `.head.parquet`, never clobbers a full build | | `--linkouts`, `-lo` | Flag | No | `False` | Append the EdgeLinkouts per-edge record URL (`https://linkouts.skyelanegoetz.com/edges/{edge_id}`, resolved to the final edge id after dedup) to the graph infores entry (`rig.source_info.infores_id`) in every emitted edge's `sources` list, so linkouts.skyelanegoetz.com can deep-link each edge; sections declaring `provenance.override` are untouched; cached to `.linkouts.parquet`, never clobbers a full build | @@ -189,8 +190,8 @@ incomplete RIG fails the build with `[rig-validation-failed]` and nothing is emi [Graph Configuration](configuration/graph.md). ??? info "Build progress & stages" - The build runs six parallel stages (Loading Tables -> Extracting Sections -> Building TCode -> - Collecting Instructions -> Building Subgraphs -> Compiling Graph) plus a seventh, Studying Graph, + The build runs six parallel stages (Loading Tables, Extracting Sections, Building TCode, + Collecting Instructions, Building Subgraphs, Compiling Graph) plus a seventh, Studying Graph, only when `--qc` is passed. They run under a three-row live progress block (stage header; section bar with count/elapsed/ETA; in-flight item detail). Each completed stage prints a green `✓ Stage N · NAME · elapsed` line above the live block. During Building diff --git a/docs/configuration/advanced-example.md b/docs/configuration/advanced-example.md index 2efb6a84..a67b889b 100644 --- a/docs/configuration/advanced-example.md +++ b/docs/configuration/advanced-example.md @@ -53,7 +53,7 @@ template: # Statistical metadata as edge annotations (method: value = constant, # method: column = per-row) annotations: - - {annotation: study_size, method: value, encoding: 9} # Study metadata -> inlined supporting Study + - {annotation: study_size, method: value, encoding: 9} # Study metadata, inlined as a supporting Study - {annotation: p_value, method: column, encoding: C} - {annotation: multiple_testing_correction_method, method: value, encoding: Benjamini Hochberg} - {annotation: effect_size, method: column, encoding: B} # Spearman rho value @@ -74,15 +74,15 @@ whole table or `method: column` to pull per-row notes from the source (see - **Excel column letters**: `encoding: A`/`B`/`C` reference the first/second/third columns of the headerless source (organism names, Spearman rho, p-value). -- **Regex pipeline**: the subject runs three substitutions in order: `.*g__` -> `` - (`d__Bacteria;p__Firmicutes;g__Lactobacillus` -> `Lactobacillus`), then `;s__` -> ` ` - (`Lactobacillus;s__rhamnosus` -> `Lactobacillus rhamnosus`), then `sp` -> `sp. `. +- **Regex pipeline**: the subject runs three substitutions in order: strip `.*g__` so + `d__Bacteria;p__Firmicutes;g__Lactobacillus` becomes `Lactobacillus`, replace `;s__` with a + space so `Lactobacillus;s__rhamnosus` becomes `Lactobacillus rhamnosus`, and expand `sp` to `sp. `. - **Taxonomic filtering**: `prioritize: [OrganismTaxon]` + `avoid: [Gene]` stop "Lactobacillus" resolving to a similarly-named gene. - **Mixed annotations**: `method: value` for constants (same every row), `method: column` for per-row values. - **Subject-predicate-object**: subject varies per row (column), predicate `correlated_with` is - fixed, object `CHEBI:41774` is fixed -> `Lactobacillus rhamnosus --[correlated_with]--> 13C-tamoxifen`. + fixed, and object `CHEBI:41774` is fixed, yielding `Lactobacillus rhamnosus --[correlated_with]--> 13C-tamoxifen`. ??? note "Regex dialect constraint" Each `pattern` is handed to Polars `str.replace_all()` (Rust `regex` crate). **Backreferences @@ -177,7 +177,7 @@ Produces two edge sets from one table: rho > 0 and rho < 0. ## Dual-Column Mapping Both subject and object come from columns, so both nodes undergo entity resolution. **Use case:** -correlation tables linking two biological entities per row (e.g., metabolite <-> microbe). +correlation tables linking two biological entities per row (for example, a metabolite and a microbe). ```yaml template: @@ -203,7 +203,7 @@ template: encoding: B # Column B: microbe names prioritize: [OrganismTaxon] regex: - - {pattern: _, replacement: ' '} # "Lactobacillus_rhamnosus" -> "Lactobacillus rhamnosus" + - {pattern: _, replacement: ' '} # "Lactobacillus_rhamnosus" becomes "Lactobacillus rhamnosus" provenance: repo: PMC @@ -216,7 +216,7 @@ template: ``` Each column-mapped node gets its own `prioritize` list to guide disambiguation. `remove` strips each -listed pattern (replace with empty string); `regex` applies an ordered `pattern`->`replacement` list; +listed pattern (replace with empty string); `regex` applies an ordered `pattern`/`replacement` list; both transform cell text in place before resolution, and neither drops rows. --- diff --git a/docs/configuration/table.md b/docs/configuration/table.md index 927fbede..76e5b2a4 100644 --- a/docs/configuration/table.md +++ b/docs/configuration/table.md @@ -269,7 +269,7 @@ annotations: - annotation: has_evidence method: column encoding: D # cells like "EFO:0001,EFO:0002" - split_by: "," # -> ["EFO:0001", "EFO:0002"], per row + split_by: "," # yields ["EFO:0001", "EFO:0002"], per row ``` The separator is a property of the data, not of the slot. Inspect the table's cells and set `split_by` to the separator the cells actually use: `","` for comma-joined ids like `"EFO:0001,EFO:0002"` above, `";"` for `"EFO:0001;EFO:0002"`, `"|"` only if the cells happen to be pipe-joined. @@ -378,7 +378,7 @@ Each entry is applied as a regex replace-with-empty-string on the cell text in p ```yaml object: encoding: identifier - prefix: "CUSTOM:" # "123" -> "CUSTOM:123" + prefix: "CUSTOM:" # "123" becomes "CUSTOM:123" ``` **Output columns: `original_` vs `_pre_resolution`** - For every subject/object/qualifier node, the pipeline snapshots the cell value into two columns at different stages: @@ -424,7 +424,7 @@ annotations: object: method: column encoding: B - explode_by: ";" # "P1;P2;P3" -> 3 separate edges + explode_by: ";" # "P1;P2;P3" becomes 3 separate edges ``` #### Mathematical Transformations @@ -480,7 +480,7 @@ Required metadata about data source. | Field | Type | Required | Description | |-------|------|----------|-------------| | `repo` | String | No | Repository. Defaults to `"PMC"`; allowed values are `"PMC"`, `"PMID"`. | -| `publication` | String | Yes, unless `override` is set | Repository-local identifier emitted as a CURIE. For `repo: PMC` the value **must** start with `PMC` followed by digits (e.g., `"PMC11708054"`, `"PMC123"`) and is emitted under the `PMCID:` namespace as `PMCID:PMC...`; for `repo: PMID` it is emitted as `PMID:` (e.g., `"11708054"` -> `PMID:11708054`). | +| `publication` | String | Yes, unless `override` is set | Repository-local identifier emitted as a CURIE. For `repo: PMC` the value **must** start with `PMC` followed by digits (e.g., `"PMC11708054"`, `"PMC123"`) and is emitted under the `PMCID:` namespace as `PMCID:PMC...`; for `repo: PMID` it is emitted as `PMID:` (e.g., `"11708054"` becomes `PMID:11708054`). | | `knowledge_level` | String | No | Biolink KL/AT knowledge level of produced edges. Defaults to `"statistical_association"`; allowed values are the `KnowledgeLevels` enum (e.g., `knowledge_assertion`, `logical_entailment`, `prediction`, `statistical_association`, `text_co_occurrence`, `observation`, `not_provided`). | | `agent_type` | String | No | Biolink KL/AT agent type responsible for produced edges. Defaults to `"data_analysis_pipeline"`; allowed values are the `AgentTypes` enum (e.g., `manual_agent`, `automated_agent`, `data_analysis_pipeline`, `computational_model`, `text_mining_agent`, `image_processing_agent`, `manual_validation_of_automated_agent`, `not_provided`). | | `override` | Object | No | Manual provenance for non-PMC/PMID sources. When set, it replaces repo/publication-derived provenance and `publication` must be omitted. | @@ -575,10 +575,10 @@ Multivalued Biolink slots such as `has_evidence` or `regulatory_approvals`, whos ```yaml annotations: - {annotation: p_value, method: column, encoding: C} # Read from column C - - {annotation: adjusted_p_value, method: column, encoding: D} # A real Association slot -> emitted on the edge - - {annotation: study_size, method: value, encoding: 450} # Study metadata -> inlined supporting Study (see below) + - {annotation: adjusted_p_value, method: column, encoding: D} # A real Association slot, emitted on the edge + - {annotation: study_size, method: value, encoding: 450} # Study metadata, inlined as a supporting Study (see below) - {annotation: multiple_testing_correction_method, method: value, encoding: "Benjamini Hochberg"} - - {annotation: has_evidence, method: column, encoding: E, split_by: ","} # cells like "EFO:0001,EFO:0002" -> a per-row JSON array + - {annotation: has_evidence, method: column, encoding: E, split_by: ","} # cells like "EFO:0001,EFO:0002" become a per-row JSON array # Descriptive name of your choice: folded into `supporting_text` on output. - annotation: log2fc_relative_to_vehicle_control @@ -638,7 +638,7 @@ Before the allow-list sweep runs, tablassert renames statistical columns to thei | Effect type | `effect_type` | `effect type`, `effect metric`, `statistic type`, `metric` | - **Negative-log p-value columns are un-logged, not shipped verbatim.** A header carrying an explicit negation marker (`negative`, `negated`, `neg`, or a leading `-`) before a `log` / `log10` p-or-q token reports a `-log10` score rather than a probability, so the value is converted back to its original scale (`p = 10 ** -x`) as the column is renamed onto `p_value` / `adjusted_p_value`, and again when the significance band is derived. Left alone, a `-log10(p)` of 8 would emit `p_value` 8.0 and band as `not_significant`, the exact inverse of the truth. The trailing token has to be a *complete* p/q-value token (or a bare delimited `P`), so a word that merely starts with p or q (`negative log protein`) is not matched. A plain `log10 p value` with no negation marker is deliberately left verbatim, because the sign convention is ambiguous there. When a raw p/q column and a `-log10` alias of the same statistic both appear, the raw column always wins selection and the alias is dropped, so a short raw name (`P`, `FDR`) cannot lose to a long alias it would then be un-logged over; the edge never carries `p_value` 0.03 beside a `"negative log10 p value: 8.0"` supporting-text entry, which reads as a contradiction rather than a duplicate. Scores extreme enough to underflow Float64 are floored at `0.0`, which is indistinguishable from `p ~ 0` in practice and lands in the same band; nulls stay null. -- **`effect_type` values are also coerced.** Each cell is matched case/separator-insensitively against an alias table (e.g. `"OR"` -> `odds_ratio`, `"Cohen's d"` -> `cohens_d`, `"Spearman"` -> `spearmans_rho`), then by `rapidfuzz` fallback against the 25 permissible `EffectTypes` values; anything matching nothing is dropped to `null` rather than carried through (the Biolink range is the enum). +- **`effect_type` values are also coerced.** Each cell is matched case/separator-insensitively against an alias table (e.g. `"OR"` maps to `odds_ratio`, `"Cohen's d"` to `cohens_d`, `"Spearman"` to `spearmans_rho`), then by `rapidfuzz` fallback against the 25 permissible `EffectTypes` values; anything matching nothing is dropped to `null` rather than carried through (the Biolink range is the enum). - **`statistical_significance_qualifier` is auto-derived** from the p-value column into five bands: `very_strongly_significant` (p <= 0.001), `strongly_significant` (<= 0.01), `significant` (<= 0.05), `suggestive` (<= 0.10), and `not_significant` (> 0.10). The values are bare `StatisticalSignificanceQualifierEnum` tokens. Biolink-model 4.4.4 attaches the slot to `Association` with the enum as its range, so the band rides the edge as a real field with no `biolink:` prefix. The same rigorous selection picks the source column: a raw `p_value` column is preferred, `adjusted_p_value` is the fallback, and the qualifier is omitted entirely when no p-value column is present. - **`effect_size` and `effect_type` travel as a pair.** A section declaring one without the other does not fail validation: the unpaired annotation is **dropped** from the section with an `UnpairedEffectAnnotationWarning` naming what was dropped and from where, and the section's edges are kept; neither half carries evidence alone. A bare effect size is uninterpretable (0.85 of *what*, an odds ratio or a Spearman rho?), and Biolink PR #1774 only populates `effect_type` alongside a numeric `effect_size`, so the build nulls an unpaired type anyway. Declare both together to retain the full evidence. Alias spellings count: `odds ratio` and the legacy `relationship_strength` coerce to `effect_size`, so both still need a sibling `effect_type`. Use `method: value` when every row shares one statistic and `method: column` when the table provides it; in a `template` + `sections` config the two lists are concatenated, so a constant `effect_type` declared once on the template pairs with each section's own `effect_size` column. - **Biolink class rules are enforced.** `effect_type` is nulled on every row where `effect_size` is null (and nulled entirely when no `effect_size` column exists); `statistical_significance_qualifier` is only set when `p_value`/`adjusted_p_value` is populated, and null p-values yield a null qualifier. diff --git a/docs/examples.md b/docs/examples.md index 3a01845d..14b2d472 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -147,7 +147,7 @@ template: **Key techniques:** -- **Regex pipeline** cleans raw taxonomic strings (`d__Bacteria;p__Firmicutes;g__Lactobacillus` -> `Lactobacillus`). Patterns must be Polars `str.replace_all()`-compatible (Rust `regex` engine): no backreferences (`\1`, `\2`, ...) or lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?postings index. +by default) holding the term-to-postings index. Fullmap is built entirely in-process by Tablassert's own Rust extension: no external tool or install step required by default (this is an in-process redb shard scheme, not the older external DuckDB shards). When the optional `[aria2]` extra is installed (`pip install "tablassert[aria2]"`; Linux/Windows wheels only), both download stages (the prebuilt `fullmap.tar.zst` archive and, for a from-scratch build, the BABEL class and synonym files) use the bundled aria2c binary automatically; without it, Tablassert's Python downloader is used. @@ -40,21 +40,22 @@ records its allowlist identity in `META.taxon_allowlist`, and that identity gate - A database already present at `--output` is reused only when its recorded identity matches; otherwise it is rebuilt. -See the [CLI Reference -> build-fullmap](cli.md#build-fullmap) for the complete flag table (output path, +See the [build-fullmap CLI reference](cli.md#build-fullmap) for the complete flag table (output path, cache directory, BABEL snapshot version, and the `--force` / `-f` rebuild flag), their defaults, and more examples. By default, `build-fullmap` first downloads a **prebuilt** database published for this Tablassert version, a `fullmap.tar.zst` under `.../fullmap//` (the version directory is the installed package version, never hardcoded), verified against a co-published `sha256sum.txt` and -extracted beside `--output` entirely in the Rust extension: it streams the archive through zstd -> tar -(the multi-GB decompressed tar is never materialized on disk) with the GIL released, extracts into a +extracted beside `--output` entirely in the Rust extension: it streams the archive through zstd +decompression and untars it (the multi-GB decompressed tar is never materialized on disk) with the +GIL released, extracts into a temp directory on the output's filesystem, and, before renaming anything into place, validates the bundle against the same contract a `--force` build must satisfy: the `meta` schema tag is exactly `tablassert.fullmap.v6`, a `build_id` is recorded, the shard files are exactly the set the primary advertises (no gaps, no extras), every shard's `build_id` equals the primary's, and the primary's `META.taxon_allowlist` identity equals the one the built-in allowlist would record. Only a bundle that -passes is atomically renamed into place (primary -> `--output`, shards beside it); any failure raises +passes is atomically renamed into place (the primary to `--output`, shards beside it); any failure raises and the command falls back to the from-scratch build below. If no prebuilt is published for this version it falls back the same way; `--force` / `-f` skips the prebuilt attempt and always builds. When the optional `[aria2]` extra is installed, the bundled aria2c from that extra accelerates @@ -79,7 +80,7 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens 2. **Equivalents index**: parse class files into sorted on-disk runs, then k-way merge them into a memory-mapped index mapping each primary CURIE to its equivalents. 3. **Synonym pass**: a producer/consumer pool streams byte-bounded line-chunks; workers dedup CURIEs, - accumulate normalized-term -> (CURIE, source) postings, and spill per-shard runs to disk so peak RAM + accumulate postings mapping each normalized term to its (CURIE, source) pairs, and spill per-shard runs to disk so peak RAM stays flat regardless of input size. 4. **Write**: one redb transaction writes the primary tables, then `shard_count` independent k-way merges write the shard `records` files in parallel (one thread per shard). @@ -98,15 +99,15 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens bar stays clean. The progress detail remains file-level (`aria2c downloading`) rather than byte-level in that mode. - **Equivalents index**: class files parse in parallel into sorted on-disk runs, k-way merged into a - single memory-mapped CURIE->equivalents index; only a compact `(hash, offset)` index lives in RAM, + single memory-mapped CURIE-to-equivalents index; only a compact `(hash, offset)` index lives in RAM, the string data is mmap'd. - **Synonym pass**: uses **intra-file parallelism**: a small pool of producer threads decompresses/reads the synonym files and pushes byte-bounded line-chunks through a bounded channel, and every worker draws from one shared queue, so the few very large files (protein/smallmolecule/gene/drugchemicalconflated) are processed by **all** workers, not one thread each. Per row, the build collects dimension sets (CURIE prefixes, Biolink categories, sources), - assigns compact integer CURIE IDs via a hash-keyed dedup map (`xxh3_128(curie) -> id`), and - accumulates normalized-term -> (CURIE, source) postings. Each worker drains its per-CURIE rows and + assigns compact integer CURIE IDs via a dedup map keyed on `xxh3_128(curie)`, and + accumulates postings mapping each normalized term to its (CURIE, source) pairs. Each worker drains its per-CURIE rows and term postings to bounded on-disk spill runs once its buffer fills, the term postings partitioned per-shard at spill time (each `run_s{shard}_{id}.bin` holds only terms with `xxh64(term) & (shards-1)` matching that shard), which keeps peak RAM flat regardless of input size @@ -134,12 +135,12 @@ override only when targeting an unusual machine. | Variable | Default | Description | |----------|---------|-------------| | `TABLASSERT_FULLMAP_EXCLUDE_PREFIXES` | *(empty)* | Comma-separated CURIE prefixes to drop at build time (e.g. `Publication`). Excluding prefixes you never resolve dramatically cuts build time, peak memory, and database size. InChIKey terms are retained unless excluded explicitly. | -| `TABLASSERT_FULLMAP_CHUNK_BYTES` | `8388608` (8 MiB) | Byte budget per producer->worker line-chunk. Bounded by bytes (not line count) so chunk memory is fixed even for large synonym records. | +| `TABLASSERT_FULLMAP_CHUNK_BYTES` | `8388608` (8 MiB) | Byte budget per producer-to-worker line-chunk. Bounded by bytes (not line count) so chunk memory is fixed even for large synonym records. | | `TABLASSERT_FULLMAP_PRODUCERS` | `clamp(workers/4, 4, #files)` | Number of producer (decompressor) threads. Decompression far outpaces parallel processing, so a handful keeps all workers fed. | -| `TABLASSERT_FULLMAP_LOCAL_SPILL_ENTRIES` | `1000000` | Per-worker term-posting buffer size before spilling a sorted run to disk. Lower -> less RAM, more run files. | -| `TABLASSERT_FULLMAP_CURIE_SPILL_ENTRIES` | `250000` | Per-worker CURIE-row buffer size before spilling to disk. Lower -> less RAM, more run files. | +| `TABLASSERT_FULLMAP_LOCAL_SPILL_ENTRIES` | `1000000` | Per-worker term-posting buffer size before spilling a sorted run to disk. Lower values use less RAM but produce more run files. | +| `TABLASSERT_FULLMAP_CURIE_SPILL_ENTRIES` | `250000` | Per-worker CURIE-row buffer size before spilling to disk. Lower values use less RAM but produce more run files. | | `TABLASSERT_FULLMAP_EQUIV_SPILL_ENTRIES` | `2000000` | Per-thread equivalents buffer size before spilling during the equivalents-index build. | -| `TABLASSERT_FULLMAP_INSERT_BATCH` | `2000000` | Records buffered per hash-sorted batch during the redb write. Larger -> faster writes, modestly more RAM. | +| `TABLASSERT_FULLMAP_INSERT_BATCH` | `2000000` | Records buffered per hash-sorted batch during the redb write. Larger batches write faster but use modestly more RAM. | | `TABLASSERT_FULLMAP_REDB_CACHE_BYTES` | `2147483648` (2 GiB) | redb write-cache size. | | `TABLASSERT_FULLMAP_SPILL_DIR` | `.spill.d` | Directory for intermediate spill runs (removed on success). | @@ -153,11 +154,11 @@ they hold six tables (see `rust/src/fullmap.rs`): | Table | Description | |-------|-------------| -| `records` | Normalized term (xxhash `u64`) -> serialized list of resolution postings (CURIE id, source id). Hash-partitioned across the shard files (`fullmap.s0..s.redb`) by `xxh64(term) & (shards-1)`; a term lives in exactly one shard. | -| `prefixes` | Compact `u16` id -> CURIE prefix string (primary file) | -| `categories` | Compact `u16` id -> Biolink category string (primary file) | -| `sources` | Compact `u8` id -> source metadata (name/version) (primary file) | -| `curies` | Compact `u32` id -> CURIE record (CURIE, preferred name, category, taxon, source) (primary file) | +| `records` | Normalized term (xxhash `u64`) mapped to a serialized list of resolution postings (CURIE id, source id). Hash-partitioned across the shard files (`fullmap.s0..s.redb`) by `xxh64(term) & (shards-1)`; a term lives in exactly one shard. | +| `prefixes` | Maps a compact `u16` id to its CURIE prefix string (primary file) | +| `categories` | Maps a compact `u16` id to its Biolink category string (primary file) | +| `sources` | Maps a compact `u8` id to its source metadata (name/version) (primary file) | +| `curies` | Maps a compact `u32` id to its CURIE record (CURIE, preferred name, category, taxon, source) (primary file) | | `meta` | Schema version tag (`tablassert.fullmap.v6`), the shard count (`shards`), the BABEL `source_version` used to build the file, and the allowlist identity (`taxon_allowlist`) recorded for every `build-fullmap` output; absent only in databases built unfiltered through the lower-level surfaces (the Rust API, or `build_fullmap_pipeline`'s `taxon_allowlist=None` default) (primary file) | The shard files must remain alongside the primary file: lookups discover them as siblings of the @@ -168,7 +169,7 @@ Lookups (`lookup_fullmap_terms`) check the primary's `meta` schema tag before re parallel (releasing the GIL, one reader per non-empty shard, re-merged into input order); a mismatched or missing tag raises rather than silently reading incompatible data. Databases built under the older `v1`/`v2`/`v3`/`v4`/`v5` schemas are rejected: there is no automatic schema migration, so a schema bump -(including the v3->v4 move to sharded files, the v4->v5 move to the redb 4 engine, and the v5->v6 move to normalized level-one keys) requires rebuilding +(the v3 to v4 move to sharded files, the v4 to v5 move to the redb 4 engine, and the v5 to v6 move to normalized level-one keys) requires rebuilding via `tablassert build-fullmap`. Readers open every fullmap file READ-ONLY with a SHARED file lock (redb >= 3 `ReadOnlyDatabase`), so any diff --git a/docs/installation.md b/docs/installation.md index 7b2f4842..f1fa93f0 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -36,7 +36,7 @@ uv tool install "tablassert[cli]" # or: pip install "tablassert[cli]" | `cli` | `tablassert` command and rich terminal progress | `cyclopts`, `rich` | | `rt` | Runtime-compatible Polars build | `polars[rtcompat]` | | `aria2` | Bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `aria2==0.0.1b0` (imports as `aria2c`, bundles aria2c) | -| `qc` | QC runtime (exact -> fuzzy -> abbreviation -> SapBERT audit) | `scikit-learn`, `sentence-transformers` (`torch` + `numpy` arrive transitively; `rapidfuzz` is a core dependency) | +| `qc` | QC runtime (exact match, fuzzy match, abbreviation expansion, then a SapBERT embedding audit) | `scikit-learn`, `sentence-transformers` (`torch` + `numpy` arrive transitively; `rapidfuzz` is a core dependency) | | `log` | loguru-backed file/progress logging (rotation, enqueue) | `loguru` | Install any extra the same way: `uv tool install "tablassert[]"` or diff --git a/docs/tutorial.md b/docs/tutorial.md index d2b3143d..c10a74db 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -158,7 +158,7 @@ tablassert build-kg tutorial-graph.yaml --qc --log - Reads the source file from disk - Applies row slicing - Resolves entities (genes and diseases) - - Validates with the QC pipeline when `--qc` is passed (exact -> fuzzy -> abbreviation -> SapBERT) + - Validates with the QC pipeline when `--qc` is passed (exact match, fuzzy match, abbreviation expansion, then SapBERT) - Creates subgraph parquet file 3. Aggregates all subgraphs 4. Exports NDJSON files and the RIG @@ -202,7 +202,7 @@ The Resource Ingest Guide records the graph's source metadata and terms of use ( ## Understanding the Transformation -Entity resolution maps text to CURIEs ("TP53" -> `HGNC:11998` Gene, "lung cancer" -> `MONDO:0008903` Disease); when `--qc` is passed, the QC pipeline validates each mapping across four stages (exact -> fuzzy -> abbreviation -> SapBERT). The output is KGX-compliant nodes and edges with standardized CURIEs, Biolink categories and predicates, provenance, and edge annotations. +Entity resolution maps text to CURIEs ("TP53" becomes the Gene `HGNC:11998`, "lung cancer" becomes the Disease `MONDO:0008903`); when `--qc` is passed, the QC pipeline validates each mapping across four stages (exact match, fuzzy match, abbreviation expansion, then SapBERT). The output is KGX-compliant nodes and edges with standardized CURIEs, Biolink categories and predicates, provenance, and edge annotations. ## Next Steps