Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,8 @@ See the [Tutorial](https://skyeav.github.io/Tablassert/tutorial/) for the full w
- **Declarative YAML configuration**: define data transformations without writing code
- **Built-in entity resolution**: map free text to genes, diseases, and chemicals with standard
CURIEs, taxonomic filtering, and provenance, backed by an embedded redb database
- **Optional quality control**: a four-stage audit (exact -> fuzzy -> abbreviation -> SapBERT embeddings) flags
low-confidence mappings
- **Optional quality control**: a four-stage audit (exact match, fuzzy match, abbreviation
expansion, then SapBERT embeddings) flags low-confidence mappings
- **KGX compliance**: emits NCATS Translator-compatible node/edge NDJSON with Biolink categories
and predicates
- **Performance & reproducibility**: lazy Polars pipelines and a deterministic UV-based
Expand All @@ -148,7 +148,7 @@ provides the `tablassert` command. Other extras are opt-in:
| `cli` | `tablassert` command and rich terminal progress | `pip install "tablassert[cli]"` |
| `rt` | CPU-compatible Polars runtime | `pip install "tablassert[rt]"` |
| `aria2` | bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `pip install "tablassert[aria2]"` |
| `qc` | four-stage QC audit (exact -> fuzzy -> abbreviation -> SapBERT embeddings) | `pip install "tablassert[qc]"` |
| `qc` | four-stage QC audit (exact match, fuzzy match, abbreviation expansion, and SapBERT embeddings) | `pip install "tablassert[qc]"` |
| `log` | loguru-backed file/progress logging (rotation, enqueue) | `pip install "tablassert[log]"` |

Reaching a feature whose extra is missing never produces a bare `ModuleNotFoundError`: the failure
Expand Down
6 changes: 3 additions & 3 deletions docs/api/fullmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ from tablassert.biolink import Categories
db = Path("/path/to/fullmap/data/fullmap.redb")
lf = pl.LazyFrame({"gene": ["TP53", "BRCA1", "EGFR", "KRAS"]})
lf = level_one(lf, "gene") # canonical tokenize, stem, deduplicate, and sort
lf = level_two(lf, "gene") # remove non-word chars -> "gene_two" column
lf = level_two(lf, "gene") # strips non-word chars into the "gene_two" column

result = resolve(lf=lf, col="gene", db=db, taxon="9606",
prioritize=[Categories.GENE], log=False).collect()
Expand All @@ -198,7 +198,7 @@ print(result.select(["gene", "gene_name", "gene_category"]))
Schema-v5 fullmap databases are rejected by the current resolver. Rebuild existing fullmaps to produce schema v6 before using them with this level-one contract.

**`level_two` output** (column: `col + "_two"`):
- All non-word characters removed (`\W+` -> `""`) from the `level_one` result
- All non-word characters (`\W+`) removed from the `level_one` result
- Used as fallback when `level_one` produces no match
- Preferred for disease names and free text

Expand All @@ -218,7 +218,7 @@ See [Quality Control](qc.md) for details.

Single-shot inspection core behind the `tablassert quick-map` CLI command: resolve raw terms
against a fullmap exactly as a build would, one result frame per input term. It runs the same op
chain a build runs per node column (normalization -> probe keys -> one batched redb fetch ->
chain a build runs per node column (normalization, probe keys, one batched redb fetch, then
filter/rank/dedup), so the returned rows are the rows `build-kg` would emit for a cell holding
that term under a `NodeEncoding` with the same settings. The whole input is ONE batched lookup,
never a round trip per term.
Expand Down
4 changes: 2 additions & 2 deletions docs/api/lib.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,8 @@ result: list[dict[str, Any]] = resolve_many(
prioritize=[Categories.GENE],
)

# result[0] -> {"original_gene": "TP53", "gene": "HGNC:11998", "gene_name": "TP53", ...}
# result[1] -> {"original_gene": "BRCA1", "gene": "HGNC:1100", "gene_name": "BRCA1", ...}
# result[0]: {"original_gene": "TP53", "gene": "HGNC:11998", "gene_name": "TP53", ...}
# result[1]: {"original_gene": "BRCA1", "gene": "HGNC:1100", "gene_name": "BRCA1", ...}
```

#### Consuming Results
Expand Down
12 changes: 6 additions & 6 deletions docs/api/qc.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ or fuzz.partial_token_sort_ratio(original, preferred) >= 80
_is_abbrev(original, preferred_name) or _is_abbrev(preferred_name, original)
```

The matcher scans the short form right-to-left against the long form (case-insensitively); the first character of the short form must land on a word boundary of the long form. This rescues the class both fuzzy matching and embedding similarity can miss: `AML` <-> `acute myeloid leukemia`.
The matcher scans the short form right-to-left against the long form (case-insensitively); the first character of the short form must land on a word boundary of the long form. This rescues the class both fuzzy matching and embedding similarity can miss, the `AML` / `acute myeloid leukemia` pair.

**Performance:** O(n) character scans per row, no model inference.

Expand Down Expand Up @@ -181,10 +181,10 @@ validated = fullmap_audit(
```
Input: 1000 rows with entity mappings

Stage 1 (Exact): 700 pass -> 300 pending
Stage 2 (Fuzzy): 250 pass -> 50 pending
Stage 3 (Abbreviation): 10 pass -> 40 pending
Stage 4 (SapBERT): 30 pass -> 10 rejected
Stage 1 (Exact): 700 pass, 300 pending
Stage 2 (Fuzzy): 250 pass, 50 pending
Stage 3 (Abbreviation): 10 pass, 40 pending
Stage 4 (SapBERT): 30 pass, 10 rejected

Output: 990 rows (700 + 250 + 10 + 30)
```
Expand All @@ -195,7 +195,7 @@ Output: 990 rows (700 + 250 + 10 + 30)
|-------|--------|-----------|----------|
| 1 | Exact match / rule-based | Highest | Standardized IDs, acronyms, CURIE-like inputs |
| 2 | Fuzzy | Medium | Typos, word reordering |
| 3 | Abbreviation expansion | High | Abbreviation <-> full-name pairs |
| 3 | Abbreviation expansion | High | Abbreviation / full-name pairs |
| 4 | SapBERT | High | Synonyms, paraphrases |

### Rejection Logging
Expand Down
15 changes: 8 additions & 7 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,9 @@ Every one of those invocations produces a database filtered by the checked-in
By default `build-fullmap` looks for a prebuilt `fullmap.tar.zst` at
`https://stars.renci.org/var/babel_outputs/<babel-version>/fullmap/<tablassert-version>/` (the version
directory is the **installed Tablassert package version**, never hardcoded), verifies it against the
published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension, streaming zstd ->
tar with the GIL released (the decompressed tar never touches disk), then validating the extracted
published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension, streaming the
archive through zstd decompression and untarring it with the GIL released (the decompressed tar never
touches disk), then validating the extracted
primary + shards against the force-build contract (exact `v6` schema, a recorded `build_id`, the exact
shard set, per-shard `build_id` equality, and a `META.taxon_allowlist` identity matching the built-in
allowlist) before atomically renaming them into place. If no prebuilt exists for this version (or the
Expand Down Expand Up @@ -103,8 +104,8 @@ See [Fullmap](fullmap.md) for the data pipeline, output schema, and graph-config

Use this to see what fullmap entity resolution will do with one or more terms (the rows a
`build-kg` run would emit for a cell holding that term under the filters shown. Each term runs
through the exact build path (level-one/level-two normalization -> probe-key extraction -> redb
fetch -> filter, rank, dedup), one `rich` table per term in input order, with the term's normalized
through the exact build path (level-one/level-two normalization, probe-key extraction, one redb
fetch, then filter, rank, and dedup), one `rich` table per term in input order, with the term's normalized
probe keys in the title so a miss is diagnosable. The whole input is one batched redb round trip,
never one per term. A term with no matches prints a `no matches` line and still exits 0: a miss is
a finding, not a failure; only usage errors and an unreadable fullmap exit 2.
Expand Down Expand Up @@ -165,7 +166,7 @@ The positional `GRAPH-CONFIGURATION-FILE` (also `--configuration-file`, `-f`) is
| --- | --- | --- | --- | --- |
| `GRAPH-CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | n/a | Graph YAML |
| `--release`, `-r` | Flag | No | `False` | Emit a slim, significant-only graph (drops `not_significant` edges before resolution) |
| `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact -> fuzzy -> abbreviation -> SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON: no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), no stray whitespace (verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies), and, when any section declares a `category_override`, no edge demoted to bare `biolink:Association` (a row escaped its pin and shipped without the class-specific slots `prune_to_class` nulled into `has_supporting_studies`; the assertion is graph-wide, so one pinned section also gates edges from unpinned sections) and fails the build (non-zero exit) on any violation |
| `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact match, fuzzy match, abbreviation expansion, then SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON: no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), no stray whitespace (verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies), and, when any section declares a `category_override`, no edge demoted to bare `biolink:Association` (a row escaped its pin and shipped without the class-specific slots `prune_to_class` nulled into `has_supporting_studies`; the assertion is graph-wide, so one pinned section also gates edges from unpinned sections) and fails the build (non-zero exit) on any violation |
| `--log`, `-l` | Flag | No | `False` | Enable verbose per-section logging; requires the `[log]` extra, and without it Tablassert produces no logs |
| `--head`, `-hd` | Flag | No | `False` | Fast output-shape preview: <=5 random rows/section, cached to `.head.parquet`, never clobbers a full build |
| `--linkouts`, `-lo` | Flag | No | `False` | Append the EdgeLinkouts per-edge record URL (`https://linkouts.skyelanegoetz.com/edges/{edge_id}`, resolved to the final edge id after dedup) to the graph infores entry (`rig.source_info.infores_id`) in every emitted edge's `sources` list, so linkouts.skyelanegoetz.com can deep-link each edge; sections declaring `provenance.override` are untouched; cached to `.linkouts.parquet`, never clobbers a full build |
Expand All @@ -189,8 +190,8 @@ incomplete RIG fails the build with `[rig-validation-failed]` and nothing is emi
[Graph Configuration](configuration/graph.md).

??? info "Build progress & stages"
The build runs six parallel stages (Loading Tables -> Extracting Sections -> Building TCode ->
Collecting Instructions -> Building Subgraphs -> Compiling Graph) plus a seventh, Studying Graph,
The build runs six parallel stages (Loading Tables, Extracting Sections, Building TCode,
Collecting Instructions, Building Subgraphs, Compiling Graph) plus a seventh, Studying Graph,
only when `--qc` is passed. They run under a three-row live progress block (stage header;
section bar with count/elapsed/ETA; in-flight item detail). Each completed
stage prints a green `✓ Stage N · NAME · elapsed` line above the live block. During Building
Expand Down
16 changes: 8 additions & 8 deletions docs/configuration/advanced-example.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ template:
# Statistical metadata as edge annotations (method: value = constant,
# method: column = per-row)
annotations:
- {annotation: study_size, method: value, encoding: 9} # Study metadata -> inlined supporting Study
- {annotation: study_size, method: value, encoding: 9} # Study metadata, inlined as a supporting Study
- {annotation: p_value, method: column, encoding: C}
- {annotation: multiple_testing_correction_method, method: value, encoding: Benjamini Hochberg}
- {annotation: effect_size, method: column, encoding: B} # Spearman rho value
Expand All @@ -74,15 +74,15 @@ whole table or `method: column` to pull per-row notes from the source (see

- **Excel column letters**: `encoding: A`/`B`/`C` reference the first/second/third columns of the
headerless source (organism names, Spearman rho, p-value).
- **Regex pipeline**: the subject runs three substitutions in order: `.*g__` -> ``
(`d__Bacteria;p__Firmicutes;g__Lactobacillus` -> `Lactobacillus`), then `;s__` -> ` `
(`Lactobacillus;s__rhamnosus` -> `Lactobacillus rhamnosus`), then `sp` -> `sp. `.
- **Regex pipeline**: the subject runs three substitutions in order: strip `.*g__` so
`d__Bacteria;p__Firmicutes;g__Lactobacillus` becomes `Lactobacillus`, replace `;s__` with a
space so `Lactobacillus;s__rhamnosus` becomes `Lactobacillus rhamnosus`, and expand `sp` to `sp. `.
- **Taxonomic filtering**: `prioritize: [OrganismTaxon]` + `avoid: [Gene]` stop "Lactobacillus"
resolving to a similarly-named gene.
- **Mixed annotations**: `method: value` for constants (same every row), `method: column` for
per-row values.
- **Subject-predicate-object**: subject varies per row (column), predicate `correlated_with` is
fixed, object `CHEBI:41774` is fixed -> `Lactobacillus rhamnosus --[correlated_with]--> 13C-tamoxifen`.
fixed, and object `CHEBI:41774` is fixed, yielding `Lactobacillus rhamnosus --[correlated_with]--> 13C-tamoxifen`.

??? note "Regex dialect constraint"
Each `pattern` is handed to Polars `str.replace_all()` (Rust `regex` crate). **Backreferences
Expand Down Expand Up @@ -177,7 +177,7 @@ Produces two edge sets from one table: rho > 0 and rho < 0.
## Dual-Column Mapping

Both subject and object come from columns, so both nodes undergo entity resolution. **Use case:**
correlation tables linking two biological entities per row (e.g., metabolite <-> microbe).
correlation tables linking two biological entities per row (for example, a metabolite and a microbe).

```yaml
template:
Expand All @@ -203,7 +203,7 @@ template:
encoding: B # Column B: microbe names
prioritize: [OrganismTaxon]
regex:
- {pattern: _, replacement: ' '} # "Lactobacillus_rhamnosus" -> "Lactobacillus rhamnosus"
- {pattern: _, replacement: ' '} # "Lactobacillus_rhamnosus" becomes "Lactobacillus rhamnosus"

provenance:
repo: PMC
Expand All @@ -216,7 +216,7 @@ template:
```

Each column-mapped node gets its own `prioritize` list to guide disambiguation. `remove` strips each
listed pattern (replace with empty string); `regex` applies an ordered `pattern`->`replacement` list;
listed pattern (replace with empty string); `regex` applies an ordered `pattern`/`replacement` list;
both transform cell text in place before resolution, and neither drops rows.

---
Expand Down
Loading
Loading