diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 265c6942..f9523ac1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -27,7 +27,7 @@ Shortcut: make setup ``` -## Daily edit → check loop +## Daily edit -> check loop ```bash # after changing Rust code, or when the extension may be stale @@ -169,9 +169,19 @@ If your OS hard limit is lower, raise the system/user NOFILE limit first, then o - Run `make check` and include the result in the PR description. - PRs are squash-merged to `main`; write commits and PR titles so the squashed history stays clear. -## Reporting issues - -Open bug reports and feature requests at [github.com/SkyeAv/Tablassert/issues](https://github.com/SkyeAv/Tablassert/issues). Include reproduction steps, the command you ran, and relevant environment details. +## Getting help + +- **Usage questions:** open an issue with the `question` label at + [github.com/SkyeAv/Tablassert/issues](https://github.com/SkyeAv/Tablassert/issues). +- **Bugs and feature requests:** use the bug-report or feature-request template in the same tracker, + with reproduction steps, the command you ran, its full output, `tablassert --version`, and your OS + and Python version. +- **Documentation drift:** the published site at + [skyeav.github.io/Tablassert](https://skyeav.github.io/Tablassert/) is the reference for the CLI, + configuration schema, and API. When prose and source disagree, `src/tablassert/models.py` and + `src/tablassert/cli.py` are the authority; file an issue for the drift. +- **Security problems:** do not open a public issue. Email the maintainers listed in + [`CITATION.cff`](CITATION.cff) so the report stays private until a fix ships. ## License diff --git a/README.md b/README.md index eef90afa..c2afa08e 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,35 @@ KGX-compliant nodes and edges. **[Full Documentation](https://skyeav.github.io/Tablassert/)**: installation guides, tutorial, configuration reference, and API docs. +## Statement of need + +Biomedical knowledge lives in spreadsheets: association tables, assay results, curated gene-disease +lists. Getting those rows into an NCATS Translator knowledge graph means writing an ingest that maps +columns to Biolink statements, resolves free text ("TP53", "lung cancer") to standard CURIEs, and +records provenance, terms of use, and statistical annotations. Today that ingest is Python code +maintained per source, and entity resolution usually means calling a hosted name-resolution service +at build time. + +Tablassert replaces the per-source code with a declarative YAML mapping plus an embedded, offline +entity-resolution database built from RENCI BABEL exports, so a build is reproducible, auditable, +and network-free. It is aimed at Translator ingest authors and knowledge-graph data engineers who +hold tabular biomedical sources, and at bioinformatics groups that want a validated KGX output +without writing a transform pipeline. The configuration model and the Biolink/KGX contracts it +emits follow current [`NCATSTranslator/translator-ingests`](https://github.com/NCATSTranslator/translator-ingests) +practice, and each build can emit the Resource Ingest Guide (RIG) metadata those submissions require. + +## Getting help + +- **Questions and bug reports:** open an issue at + [github.com/SkyeAv/Tablassert/issues](https://github.com/SkyeAv/Tablassert/issues) (use the + `question` label for usage questions). Include the command you ran, its output, and your + `tablassert --version`. +- **Contributing code or docs:** see [CONTRIBUTING.md](CONTRIBUTING.md) for setup, the local quality + gates, and pull-request expectations. +- **Reference:** the [CLI Reference](https://skyeav.github.io/Tablassert/cli/) and + [configuration guides](https://skyeav.github.io/Tablassert/configuration/graph/) document every + flag and field. + ## Quick Start ```bash @@ -44,14 +73,17 @@ template: ``` Wrap it in a graph config (`graph.yaml`) pointing at your fullmap entity-resolution database -and carrying the required `rig:` metadata for the generated Resource Ingest Guide: +and carrying the required `rig:` metadata for the generated Resource Ingest Guide. Build or +download that database once first (`tablassert build-fullmap`, a multi-GB download; see the +[Fullmap guide](https://skyeav.github.io/Tablassert/fullmap/)), which makes `./fullmap` below the +right path: ```yaml name: MY_KG version: 1.0.0 tables: - ./table.yaml -fullmap: /path/to/fullmap +fullmap: ./fullmap rig: source_info: infores_id: infores:my-kg @@ -98,7 +130,8 @@ See the [Tutorial](https://skyeav.github.io/Tablassert/tutorial/) for the full w low-confidence mappings - **KGX compliance**: emits NCATS Translator-compatible node/edge NDJSON with Biolink categories and predicates -- **Autonomous agent**: `tablassert agent` derives, builds, and refines configs for whole papers +- **Autonomous agent (experimental)**: `tablassert agent` derives, builds, and refines configs for + whole papers - **Performance & reproducibility**: lazy Polars pipelines and a deterministic UV-based development environment @@ -108,8 +141,8 @@ See the [Tutorial](https://skyeav.github.io/Tablassert/tutorial/) for the full w pip install tablassert ``` -Or with uv: `uv tool install "tablassert[cli]"`. The base install provides the Python API; -install `[cli]` to use the `tablassert` command and optional extras for additional runtime and pipeline capabilities: +Or with uv: `uv tool install "tablassert[cli]"`. The base install provides the Python API; `[cli]` +provides the `tablassert` command. Other extras are opt-in: | Extra | Adds | Install | | ----- | ---- | ------- | @@ -117,17 +150,16 @@ install `[cli]` to use the `tablassert` command and optional extras for addition | `rt` | CPU-compatible Polars runtime | `pip install "tablassert[rt]"` | | `aria2` | bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `pip install "tablassert[aria2]"` | | `qc` | four-stage QC audit (exact -> fuzzy -> abbreviation -> SapBERT embeddings) | `pip install "tablassert[qc]"` | -| `agent` | autonomous agent (smolagents, litellm, article/table context) | `pip install "tablassert[agent]"` | -| `optimize` | GEPA prompt optimization for `agent --optimize` (dspy) | `pip install "tablassert[optimize]"` | -| `distill` | distillation dataset export (`tablassert distill-export`, HF `datasets`) | `pip install "tablassert[distill]"` | +| `agent` | autonomous agent (smolagents, litellm, article/table context) -- experimental, API may change | `pip install "tablassert[agent]"` | +| `optimize` | GEPA prompt optimization for `agent --optimize` (dspy) -- experimental, API may change | `pip install "tablassert[optimize]"` | +| `distill` | distillation dataset export (`tablassert distill-export`, HF `datasets`) -- experimental, API may change | `pip install "tablassert[distill]"` | | `log` | loguru-backed file/progress logging (rotation, enqueue) | `pip install "tablassert[log]"` | -The `tablassert` command requires `[cli]`; without it, the console launcher reports the exact install command. QC is opt-in at build time (`build-kg --qc`). Reaching a feature whose extra is not installed never -produces a bare `ModuleNotFoundError`: the failure names the missing package and the exact install -command, and for `build-kg --qc` and `tablassert agent` it arrives before the run starts rather than -partway through. Logging is the exception: without the `log` extra Tablassert produces no logs -instead of failing. See the -[Installation guide](https://skyeav.github.io/Tablassert/installation/) for the full matrix and the +Reaching a feature whose extra is missing never produces a bare `ModuleNotFoundError`: the failure +names the missing package and the install command that fixes it. QC is opt-in at build time +(`build-kg --qc`). See the +[Installation guide](https://skyeav.github.io/Tablassert/installation/) for the full matrix, +per-command preflight behavior, and the [CLI Reference](https://skyeav.github.io/Tablassert/cli/) for every flag. ## Entity Resolution API diff --git a/docs/agent.md b/docs/agent.md index b2b676bd..fc7ea8ad 100644 --- a/docs/agent.md +++ b/docs/agent.md @@ -1,9 +1,14 @@ -# Autonomous Agent (`[agent]` extra) +# Autonomous Agent (`[agent]` extra, experimental) + +!!! warning "Experimental" + Everything on this page is experimental: the `tablassert agent` flags, the state-directory + layout, the tools, and the prompt-optimization/distillation surfaces may change without notice + or a deprecation cycle. The core build pipeline is unaffected and stays stable. **Why this exists:** hand-authoring a Tablassert config for every PMC supplementary table does not scale. The optional `[agent]` extra makes it autonomous: point it at **PubMed Central (PMC)** article IDs and it -**derives the config for you**, then builds, audits, and iteratively improves the graph — the -improve loop is deterministic supervisor Python, not more LLM calls — until the entity +**derives the config for you**, then builds, audits, and iteratively improves the graph (the +improve loop is deterministic supervisor Python, not more LLM calls) until the entity resolution *maps* (coverage threshold). The outcome is an **NCATS Translator-compliant KGX knowledge graph** per article, a claim the loop verifies rather than asserts, by constructing every emitted record as its own Biolink class (see [Biolink validity](#biolink-validity)), with the whole loop @@ -57,13 +62,13 @@ failing fast (cheap checks before any large download and before any model call): 1. Enumerates version prefixes via S3 `list-objects-v2` (`?list-type=2&prefix=PMC.&delimiter=/`) and selects the **latest** version (numeric, so `PMC.10` beats `PMC.2`); older versions are ignored. 2. Checks the latest version's `.json` metadata for open access (`is_pmc_openaccess` / a `CC*` - `license_code`), **before** any large download (not open access ⇒ `PermissionError` immediately). + `license_code`), **before** any large download (not open access => `PermissionError` immediately). 3. Enumerates the version's objects (`?list-type=2&prefix=PMC./`) and confirms a data table is - present (a file with extension `.xlsx .xls .csv .tsv`), **before** any large download (none ⇒ + present (a file with extension `.xlsx .xls .csv .tsv`), **before** any large download (none => `FileNotFoundError`). 4. Downloads only the **useful** files to `outdir//` and returns their paths: the main text (`.xml`/`.nxml`/`.txt`), the `.json` metadata, and every data table. Binary media (images, - `.docx`, the article `.pdf` — every version ships JATS `.xml`, so the PDF is redundant) are skipped. + `.docx`, the article `.pdf`; every version ships JATS `.xml`, so the PDF is redundant) are skipped. `fetch_pmc_tables` remains as a thin wrapper returning only the table files. The main text and every candidate table are wired into the agent TWICE, deliberately: the supervisor @@ -73,7 +78,7 @@ per-column **`column_digest`** block per previewed table/worksheet directly into (`render_task_context`), so the agent can author a config with **zero inspection tool calls**. Each digest scans the first **500 data rows** and reports, per column, the **fraction of non-null cells containing each separator** (`;`, `|`, `,`, `/`) plus non-null/distinct counts, max cell length, and -sample values — enough for `explode_by`/`split_by` detection without a single tool call. Previews and +sample values: enough for `explode_by`/`split_by` detection without a single tool call. Previews and digests are rendered inside the data fences (untrusted data, never instructions; see [Prompt-injection defenses](#prompt-injection-defenses)). The `pmc_article_context` / `read_table` tools stay registered as fallbacks for rows beyond a preview or a @@ -89,12 +94,12 @@ in the config). Small tables and worksheets are filtered before this context is !!! note "Coverage + licensing" Only the **open-access subset** of PMC (~half) is available here. Articles are **CC-BY**: cite the - source and DOI (e.g. PMC11708054 → [10.1128/mbio.01679-24](https://doi.org/10.1128/mbio.01679-24)). + source and DOI (e.g. PMC11708054 -> [10.1128/mbio.01679-24](https://doi.org/10.1128/mbio.01679-24)). ## Network resilience -Every network call the agent makes — the PMC-AWS object listing, article metadata, and file downloads -— routes through one stdlib-only seam, `tablassert.net`, which classifies each failure as transient +Every network call the agent makes (the PMC-AWS object listing, article metadata, and file downloads) +routes through one stdlib-only seam, `tablassert.net`, which classifies each failure as transient (worth retrying) or permanent (fail fast) and retries the transient ones with bounded jittered backoff. The fleet run that motivated this lost 2,194 of 2,507 queue failures (87.5%) to DNS-shaped errors (`[Errno -2] Name or service not known`): single-attempt fetches turned a recoverable @@ -106,13 +111,13 @@ resolver blip into a terminal `SKIPPED` for the whole article. | Connection loss (`ConnectionError`, `ssl.SSLError`, `http.client.HTTPException`) | `PermissionError` (metadata not CC-licensed), `FileNotFoundError` (no OA versions / no table files) | | Timeouts (`TimeoutError`; `socket.timeout` *is* `TimeoutError` on Python 3.10+) | Any 4xx except 408 / 425 / 429 | | HTTP 408 / 425 / 429 and every 5xx | A bare `OSError` with a local errno (`ENOSPC`, `EACCES`, `ENOENT`) | -| A bare `OSError` with a network errno (`ECONNRESET`, `EPIPE`, `ENETUNREACH`, `EMFILE`, …) | | -| Rate-limit / quota errors from optional libraries (`openai`, `litellm`, `httpx`), matched by exception **name** — never imported — or by a `rate limit` / `429` / `too many requests` message token; a `(reset after …)` hint then sets the wait, not the classification | | +| A bare `OSError` with a network errno (`ECONNRESET`, `EPIPE`, `ENETUNREACH`, `EMFILE`, ...) | | +| Rate-limit / quota errors from optional libraries (`openai`, `litellm`, `httpx`), matched by exception **name** (never imported) or by a `rate limit` / `429` / `too many requests` message token; a `(reset after ...)` hint then sets the wait, not the classification | | The HTTP retry budget is 4 attempts per call, sleeping 1.0 s before the second attempt and doubling per attempt up to 20.0 s, with the **sum** of sleeps inside one call capped at 60.0 s. When the failure carries a `Retry-After` header (or a `(reset after 90s)` quota hint in its message), the -longer of that hint and the jittered doubling step is slept — the hint overrides the schedule only +longer of that hint and the jittered doubling step is slept; the hint overrides the schedule only when it is longer, capped at 60 s. The worst case for one HTTP round trip is therefore 4 x 120 s timeout + 60 s backoff = 540 s before the call raises for good; the article is then `SKIPPED` with `error_code: network-transient` (see @@ -120,24 +125,24 @@ article is then `SKIPPED` with `error_code: network-transient` (see Article file downloads are idempotent, atomic, and bounded-parallel: a file that already exists non-empty is skipped **without any request** (a torn zero-length file re-downloads), each download -writes a `.part` sibling that is atomically `os.replace`d into place on completion — and removed -even on `KeyboardInterrupt`, so no half-written file survives — and up to 8 files fetch concurrently +writes a `.part` sibling that is atomically `os.replace`d into place on completion, and removed +even on `KeyboardInterrupt`, so no half-written file survives, and up to 8 files fetch concurrently by default, with results collected in submission order. `concurrency < 1` fails before any network call is made. **Exactly one retry layer exists for LLM calls.** Smolagents' own rate-limit retryer is disabled (`retry=False`) and the OpenAI client's transport retries are off (`max_retries: 0`); the shared -seam retries transient LLM failures — DNS, timeouts, 429, 5xx — with the same 4 attempts but a +seam retries transient LLM failures (DNS, timeouts, 429, 5xx) with the same 4 attempts but a 45-second per-call backoff budget. LiteLLM's internal backend retry is not exposed by its constructor and remains outside this control. -**Worst-case wall clock.** The fleet forensic run recorded ~13–18 HTTP round trips per article +**Worst-case wall clock.** The fleet forensic run recorded ~13-18 HTTP round trips per article (3 sequential listing/metadata calls plus one per downloaded file). At the per-call worst case of 540 s (which includes the 120 s socket timeout on each attempt, not just the backoff), three serial calls plus ceil(15/8) = 2 parallel download waves bound the fetch phase at 5 x 540 s = 2,700 s = 45 min; the LLM layer adds at most 29 logical calls x 45 s = 1,305 s ~ 21.8 min. Together that is ~67 min of the 90-minute per-article budget, leaving ~23 min for the -real work of deriving, building, and auditing configs — and a fully-down article never hangs a +real work of deriving, building, and auditing configs, and a fully-down article never hangs a worker: every call either succeeds, retries within budget, or raises, and an exhausted article is skipped with `network-transient` for requeue. @@ -232,33 +237,33 @@ control flow over agentic decisions. For each PMC id it: `SKIPPED` before constructing the inner model. 2. Runs the **inner `CodeAgent`** to *derive* an initial table config. The task already contains the article summary, head previews, and `column_digest` separator statistics of every table/worksheet, - so the canonical path is a fixed **derive → build → answer** workflow over the four-tool surface - (`derive_config` → `build_and_audit` → final answer, target: 3 steps or fewer); `read_table` / + so the canonical path is a fixed **derive -> build -> answer** workflow over the four-tool surface + (`derive_config` -> `build_and_audit` -> final answer, target: 3 steps or fewer); `read_table` / `pmc_article_context` remain fallbacks only for rows beyond a digest's 500-row scan window. The - agent rebuilds **only on a coded build error** — fixing exactly the field the error names, at most - twice — and never loops on coverage: coverage improvement is the supervisor's job (step 4). Every + agent rebuilds **only on a coded build error**, fixing exactly the field the error names, at most + twice, and never loops on coverage: coverage improvement is the supervisor's job (step 4). Every section is gated by the Section JSON schema. The agent maps **each** mappable table/worksheet as its own section, **one config per paper** (see below). -3. **Builds + audits** in one deterministic mega-tool (`build_and_audit`: validate → build → QC → coverage - → **Biolink validity**). The LLM sees a **compact** observation — exactly the 12 high-signal keys +3. **Builds + audits** in one deterministic mega-tool (`build_and_audit`: validate -> build -> QC -> coverage + -> **Biolink validity**). The LLM sees a **compact** observation, exactly the 12 high-signal keys (verdict, coded errors + codes, coverage/Biolink/demoted-edge scores, `predicate_advice`, `multivalued_suspects`, node/edge counts, the `head` flag, and `unresolved` capped at 20 entries - with a visible `+N more` marker) — while the pure `build_and_audit` function still hands the + with a visible `+N more` marker), while the pure `build_and_audit` function still hands the supervisor the **full** report (artifact paths, bookkeeping, strict-Biolink internals). The report is *actionable*, not just a score: a nonzero `demoted_edge_pct` comes with `predicate_advice` (the legal predicates for the demoted category pair), unresolved terms that still contain a separator surface as `multivalued_suspects` (a missed `explode_by`), and every report carries a `head` fidelity flag so sampled edge counts are never compared against full builds. -4. **Improves** coverage with **deterministic Python — never the LLM** — while coverage `< map_threshold` +4. **Improves** coverage with **deterministic Python** (never the LLM) while coverage `< map_threshold` and budget remains: tier 1 feeds `map_coverage` feedback (called as a pure function) to a **ranked** list of distinct `propose_config_edit` candidates (also pure), scores them with fast head builds, and - accepts the first full build that is **strictly better — iff no worse on coverage *or* Biolink - validity and strictly better on one** (monotonic: - regressions on either axis are rejected, so a coverage win can no longer be bought with invalid KGX) — + accepts the first full build that is **strictly better** (iff no worse on coverage *or* Biolink + validity and strictly better on one; monotonic, so + regressions on either axis are rejected and a coverage win can no longer be bought with invalid KGX), and an edit that shrinks the full-build **edge count** by more than 25% is rejected even with a gain (the detail-first objective: the biggest solid config wins). The deterministic proposer covers **four** knob families: NodeEncoding knobs (`prioritize`/`avoid`/`regex`/`remove`/`exclude_*`), - **`explode_by`** (added when unresolved terms still carry a separator), and — fed the audit report — + **`explode_by`** (added when unresolved terms still carry a separator), and, fed the audit report, a **demoted-predicate fix** (the first legal predicate from `predicate_advice`). Only tier 2, the OPT-IN `--reflexion` path, spends an LLM call, and only after tier 1 stalls; it may additionally change qualifiers, `split_by`, node categories, and the source. @@ -289,22 +294,22 @@ endpoint; neither is required): [Biolink validity](#biolink-validity) below. Both LLM-facing prompts are hard-bounded so a pathological article cannot outgrow the model's -context window. The reflexion prompt caps its three interpolated blocks — current config at 8,000 +context window. The reflexion prompt caps its three interpolated blocks (current config at 8,000 chars (`MAX_PROMPT_CONFIG_CHARS`), coverage report at 8,000 chars (`COVERAGE_PROMPT_CHARS`), -article/table context at 40,000 chars (`MAX_PROMPT_CONTEXT_CHARS`) — and the judge prompt +article/table context at 40,000 chars (`MAX_PROMPT_CONTEXT_CHARS`); and the judge prompt serializes the compact audit report bounded at 8,000 chars (`MAX_PROMPT_REPORT_CHARS`). Every `unresolved` term list keeps only its first 20 entries (`UNRESOLVED_CAP`) plus a visible `+N more` marker and an `unresolved_count` naming the original length, so truncation is never silent and the scale signal survives; the worst-case prompt is ~58,000 chars, about 15,000 tokens. The fleet distill telemetry that motivated this recorded reflexion prompts of 1.6M and 2.3M characters (689,241 and -524,336 input tokens) — past every model's window, failing outright after paying for the +524,336 input tokens) past every model's window, failing outright after paying for the serialization. Compaction happens only at prompt serialization: `map_coverage`, `build_and_audit`, and the supervisor's deterministic proposer still see the full uncapped reports. ### Distilling a fine-tuning dataset (`--distill`) -`--distill` (short: `-d`, `-dt`) records **every LLM call of the run** — the inner agent's -multi-turn conversations, plus the judge and reflexion calls when those gates are enabled — as one +`--distill` (short: `-d`, `-dt`) records **every LLM call of the run**: the inner agent's +multi-turn conversations, plus the judge and reflexion calls when those gates are enabled, as one ChatML JSON object per line, appended to `/distill/records.ndjson`. Every record line carries the same canonical 13-key v2 set, with an explicit `null` where a value is unknown, so a key never first appears partway down an append-only file: @@ -315,7 +320,7 @@ key never first appears partway down an append-only file: Alongside the records, the supervisor appends **one outcome line per run** to a sibling `outcomes.ndjson` in the same directory: the terminal status, the build/audit figures, the -tool-call tallies, the gate thresholds, and the package versions — the full 35-key outcome set, +tool-call tallies, the gate thresholds, and the package versions (the full 35-key outcome set), schema-uniform with explicit nulls in the same way (abbreviated here): ```json @@ -324,8 +329,8 @@ schema-uniform with explicit nulls in the same way (abbreviated here): `run_id` (`:`) is the join key between the two files. They are separate because the two halves exist at different times: a record is appended the moment each model call -completes, mid-run, while the outcome — build verdict, coverage, Biolink validity, demoted-edge -fraction — is only known once the supervisor has decided the run's terminal status, so it is +completes, mid-run, while the outcome (build verdict, coverage, Biolink validity, demoted-edge +fraction) is only known once the supervisor has decided the run's terminal status, so it is written exactly once, at the end of the run. Both files are **append-only** and never rewritten in place: a corpus accumulates over many batches, and retuning a weight never requires re-recording it. The `messages` column is plain ChatML, which Unsloth Studio auto-detects on JSONL upload (no @@ -339,39 +344,39 @@ training row per input record plus a reproducibility manifest. Flags: `--distill `--out`/`-o` (both required), `--policy`/`-p` (default `threshold`), `--threshold`/`-t` (`0.75`), `--top-n`/`-tn` (`2`), `--replication-k`/`-rk` (`2`), `--reward-config`/`-rc` (a YAML/JSON policy override), `--edge-ref` (a breadth-reference override), `--purpose` (default `agent`; the literal -`all` disables filtering), `--final-call-only` (keep only each run's highest `call_index` — the +`all` disables filtering), `--final-call-only` (keep only each run's highest `call_index`: the most complete conversation), and `--manifest` (default `.manifest.json`). Keep `--out` **outside** `--distill-dir`: [`tablassert distill-export`](cli.md#distill-export) loads every `*.ndjson` in its input directory, so a weighed file placed there would be re-ingested as raw -corpus. The weigh → export composition (export requires the `[distill]` extra): +corpus. The weigh -> export composition (export requires the `[distill]` extra): ```bash tablassert distill-weigh --distill-dir .tablassert/agent/distill --out ./training/train.ndjson tablassert distill-export --distill-dir ./training --out ./hf-dataset ``` -**The reward** is a deterministic function of the captured outcome — same outcome in, same weight -out — and it is **tunable policy with documented defaults**, not a law: every knob is a +**The reward** is a deterministic function of the captured outcome: same outcome in, same weight +out. It is **tunable policy with documented defaults**, not a law: every knob is a `RewardConfig` field overridable via `--reward-config`, and retuning never touches the corpus. The defaults are Tablassert's own policy (they mirror the shape of the supervisor's quality score with its two degenerate terms replaced), not a borrowed standard. Five additive terms whose coefficients sum to 1.00: -- `coverage_pct` — completeness / entity-resolution success — **0.40** -- `biolink_valid_pct` — semantic validity of the built edges — **0.28** -- `specificity = 1 - demoted_edge_pct` — predicate specificity vs generic fallback — **0.17** -- `cleanliness = 1 - (failed + wrong) / total tool calls` — tool-call correctness — **0.07** -- `breadth = clamp(log1p(edge_count) / log1p(edge_ref), 0, 1)` — non-degeneracy — **0.08** +- `coverage_pct` (completeness / entity-resolution success): **0.40** +- `biolink_valid_pct` (semantic validity of the built edges): **0.28** +- `specificity = 1 - demoted_edge_pct` (predicate specificity vs generic fallback): **0.17** +- `cleanliness = 1 - (failed + wrong) / total tool calls` (tool-call correctness): **0.07** +- `breadth = clamp(log1p(edge_count) / log1p(edge_ref), 0, 1)` (non-degeneracy): **0.08** The hard gates then apply **multiplicatively**, never additively: `ok is not True`, `head is True`, or a `SKIPPED`/`FAILED` status zero the row outright; `BUILT_UNMEASURED` (or coverage never -measured) floors the row at `unmeasured_weight` (default `0.0`) — the record is kept, just not +measured) floors the row at `unmeasured_weight` (default `0.0`); the record is kept, just not selectable. On top of the raw score, two farming penalties multiply: `demoted_edge_pct > 0.50` scales by **0.5** (generic-predicate farming), and a redundant tool-call share above **0.30** scales by **0.7**. The breadth reference `edge_ref` defaults to the **corpus median** edge count over comparable builds (head builds and failed builds excluded, deduplicated to the last outcome per run id); `--edge-ref` or the config file pins it instead, and a corpus with no comparable -build warns and lets breadth contribute 0.0. The reward reads only deterministic fields — never +build warns and lets breadth contribute 0.0. The reward reads only deterministic fields, never the judge score, `qc_pass_rate`, or `provenance_ok`, which are recorded as metadata only. **Selection policies.** All three are computed at weigh time (re-recording is never needed) and @@ -380,37 +385,37 @@ annotate every row with the same five keys (`weight`, `selected`, `replicas`, `p | Policy | What it does | Prefer when | Caveat | | --- | --- | --- | --- | -| `threshold` (default) | `selected = weight >= --threshold` (0.75); each selected row counts once | The zero-code path — the only selection TRL consumes with no trainer code | A miscalibrated corpus can empty the selection; the manifest's `selected_count` makes that visible | -| `best-of-n` | Groups by `pmc_id`, ranks each group by weight (ties: fewer attempts, fewer failed tool calls, earlier call), keeps the top `--top-n` (2) | Several attempts per article: keep the best trajectory per prompt while preserving distinct paths | **No weight floor**: a prompt whose every trajectory weighs 0.0 still contributes its top-ranked row, so gated-failed trajectories can be selected — the manifest's `selected_zero_weight` exposes this | -| `replication` | `selected = weight > 0`; `replicas` scales from 1 to `1 + --replication-k` (default 2, capped at 3) across the selected rows' weight spread | Soft importance weighting with no custom loss code | `replicas` is a **count on one row**, never physical duplication — the training sampler must expand the rows | +| `threshold` (default) | `selected = weight >= --threshold` (0.75); each selected row counts once | The zero-code path (the only selection TRL consumes with no trainer code) | A miscalibrated corpus can empty the selection; the manifest's `selected_count` makes that visible | +| `best-of-n` | Groups by `pmc_id`, ranks each group by weight (ties: fewer attempts, fewer failed tool calls, earlier call), keeps the top `--top-n` (2) | Several attempts per article: keep the best trajectory per prompt while preserving distinct paths | **No weight floor**: a prompt whose every trajectory weighs 0.0 still contributes its top-ranked row, so gated-failed trajectories can be selected; the manifest's `selected_zero_weight` exposes this | +| `replication` | `selected = weight > 0`; `replicas` scales from 1 to `1 + --replication-k` (default 2, capped at 3) across the selected rows' weight spread | Soft importance weighting with no custom loss code | `replicas` is a **count on one row**, never physical duplication: the training sampler must expand the rows | -**The goal is LoRA/QLoRA supervised fine-tuning — explicitly not RLHF.** There is no reward model, +**The goal is LoRA/QLoRA supervised fine-tuning (explicitly not RLHF).** There is no reward model, no PPO/GRPO, and no online RL anywhere in this pipeline; the reward is a deterministic scoring function used for data selection. The weight drives row filtering, ranking, and replication -because TRL's `SFTConfig` has **no per-example sample-weight column** — selection and replication +because TRL's `SFTConfig` has **no per-example sample-weight column**: selection and replication are the only zero-code weighting mechanisms an SFT trainer offers. Consumption caveats that are easy to get wrong: - `SFTConfig.max_length` defaults to **1024** with `truncation_mode="keep_start"`: long multi-turn agent trajectories are silently truncated, and TRL then **drops** examples left fully masked. - Size it from the recorded `n_messages`/`tokens_total` metadata — set `max_length=None` or to at + Size it from the recorded `n_messages`/`tokens_total` metadata; set `max_length=None` or to at least the corpus's p99. -- `assistant_only_loss=True` trains on the assistant (agent) turns only — usually what you want +- `assistant_only_loss=True` trains on the assistant (agent) turns only (usually what you want) for a config-authoring agent. - `packing=True` makes any effective weight **token-proportional rather than row-proportional** and destroys per-example identity, which quietly undermines replication. - Extra metadata columns (every `outcome_*` column, `weight`, `selected`, and friends) are - **ignored** by TRL, not fatal — safe to keep them in the file. + **ignored** by TRL, not fatal; safe to keep them in the file. Honest limitations. SFT on curated optimal trajectories stabilizes output **format and schema -compliance** — for a YAML-config-writing agent whose supervisor is deterministic Python, that is +compliance**: for a YAML-config-writing agent whose supervisor is deterministic Python, that is the honest deliverable of a LoRA. It is **not** evidence-backed for out-of-distribution generalization (Chu et al., arXiv:2501.17161, App. C.1); do not promise OOD gains. Two reward-hacking caveats: `demoted_edge_pct` gating exists because coverage can be **farmed with a -generic predicate** — a predicate the derived association class forbids never raises, it silently -demotes the edge, so a config can map perfectly while emitting bare `biolink:Association` edges — -and over-hard selection is a measured Goodhart risk (Gao et al., arXiv:2210.10760). Hold out a +generic predicate**: a predicate the derived association class forbids never raises; it silently +demotes the edge, so a config can map perfectly while emitting bare `biolink:Association` edges. +Over-hard selection is a measured Goodhart risk (Gao et al., arXiv:2210.10760). Hold out a differently-scored validation set rather than trusting the same weight that selected the training rows. @@ -428,8 +433,8 @@ the same check [`tablassert validate-kgx`](cli.md#validate-kgx) runs, and the sa | `biolink_valid_pct_strict` | Pass rate with no exemptions, so the pending gap stays visible | | `biolink_problems` | Top `"field: error-type"` failures with counts, for self-correction | | `demoted_edge_pct` | Fraction of edges that fell back to bare `biolink:Association` | -| `predicate_advice` | Per demoted (predicate, subject, object) group: the derived association class and the **legal predicates** — the exact fix, not just the symptom | -| `multivalued_suspects` | Entity columns whose unresolved terms still contain a separator (`;`, `\|`, `,`) — a missed `explode_by`, with the literal separator to declare | +| `predicate_advice` | Per demoted (predicate, subject, object) group: the derived association class and the **legal predicates**: the exact fix, not just the symptom | +| `multivalued_suspects` | Entity columns whose unresolved terms still contain a separator (`;`, `\|`, `,`): a missed `explode_by`, with the literal separator to declare | **`demoted_edge_pct` is the predicate signal.** Tablassert derives an edge's association class from the (subject category, object category) pair, then `resolve_association_class` gives up as much of @@ -445,8 +450,8 @@ cannot drift from the model the build validates against: ```text - Gene ~ Disease -> GeneToDiseaseAssociation: affects, associated_with, contributes_to -- SequenceVariant ~ Gene -> VariantToGeneAssociation: condition_associated_with_gene, … -- any predicate is safe for: Gene~Gene, Gene~Pathway, ChemicalEntity~Disease, … +- SequenceVariant ~ Gene -> VariantToGeneAssociation: condition_associated_with_gene, ... +- any predicate is safe for: Gene~Gene, Gene~Pathway, ChemicalEntity~Disease, ... ``` - An annotation like `supporting_study_size` or `sample_size` names study-level metadata. @@ -510,14 +515,14 @@ and unrelated table entries are preserved. Before the accepted best config is persisted it is also **compacted deterministically** (`compact_config`), after normalization: provably no-op entries (keys equal to the Pydantic model -defaults) are removed while semantics are preserved — the compacted config builds the identical KGX +defaults) are removed while semantics are preserved: the compacted config builds the identical KGX and scores the identical `quality_score` (pinned by the offline accuracy-invariance test). Compaction can only shrink a config or leave it alone, never corrupt it: any failure writes the normalized -uncompacted config and the status is unaffected. Each record tracks `config_chars` — the character -count of what was actually written to `configs/.yaml` — in `state.json`, so size deltas are +uncompacted config and the status is unaffected. Each record tracks `config_chars` (the character +count of what was actually written to `configs/.yaml`, in `state.json`), so size deltas are auditable per article. -Each record also carries `error_code` — the stable kebab-case `code` of the exception that caused a +Each record also carries `error_code` (the stable kebab-case `code` of the exception that caused a `SKIPPED` record, or `null` for a deterministic gate, an uncoded error, or a pre-field `state.json`. It describes the **most recent attempt**: starting an attempt clears it, so a later `MAPPED` result cannot retain a stale transient code. Consumers should inspect it only when `status == "SKIPPED"`. @@ -562,8 +567,8 @@ another's entries and a same-PMC rerun has deterministic last-writer-wins replac ## The tools -Full mode registers **exactly four** LLM tools — `read_table`, `pmc_article_context`, `derive_config`, -`build_and_audit` — the derive → build → answer surface. `map_coverage` and `propose_config_edit` are +Full mode registers **exactly four** LLM tools: `read_table`, `pmc_article_context`, `derive_config`, +`build_and_audit`: the derive -> build -> answer surface. `map_coverage` and `propose_config_edit` are **not** in the full-mode agent's surface: they are pure helpers the deterministic supervisor calls itself in its improve loop (coverage improvement is the supervisor's job, after the agent answers). Two batch derive modes vary the surface: `derive_only` registers only the three @@ -577,7 +582,7 @@ can pick the best sheet/columns). | `pmc_article_context` | tool | parse the JATS main text into a **data-fenced** summary (title/abstract/sections/supplementary manifest); `.txt` renders a fenced excerpt | | `read_table` | tool | render a table as **data-fenced, spotlighted** text; lists **all worksheets** of an Excel file (`sheet=`) | | `derive_config` | tool | author a table config (`template` + one section per table); each section must satisfy `Section.model_json_schema()` | -| `build_and_audit` | tool | **one** deterministic validate→build→QC→coverage→**Biolink-validity** mega-tool; the LLM observation is the **compact** 12-key report (`unresolved` capped at 20 + `+N more`), while direct/supervisor callers of the pure function get the **full** report; the report's `predicate_advice` / `multivalued_suspects` fields make demotions and missed `explode_by`s directly actionable | +| `build_and_audit` | tool | **one** deterministic validate->build->QC->coverage->**Biolink-validity** mega-tool; the LLM observation is the **compact** 12-key report (`unresolved` capped at 20 + `+N more`), while direct/supervisor callers of the pure function get the **full** report; the report's `predicate_advice` / `multivalued_suspects` fields make demotions and missed `explode_by`s directly actionable | | `map_coverage` | tool (`derive_coverage` mode only) / supervisor pure helper | fullmap term-resolution coverage (per-column + overall); in full mode ONLY the deterministic supervisor calls the pure function (improve loop + per-section recording), never the LLM | | `propose_config_edit` | supervisor pure helper | deterministic, constrained edits + rationale used ONLY by the supervisor's improve loop (never an LLM tool): `NodeEncoding` knobs, `explode_by` from separator-carrying unresolved terms, and (given the audit report) a demoted-predicate fix | @@ -590,25 +595,25 @@ gate can only answer true/false and would otherwise swallow the reason. The agent's `instructions` make the techniques explicit: -- **Detail-first goal ordering**: the goals are (1) BREADTH + DETAIL — every mappable sheet as its own +- **Detail-first goal ordering**: the goals are (1) BREADTH + DETAIL (every mappable sheet as its own section, every evidence slot captured, multi-valued cells exploded, direction/aspect columns - qualified; (2) coverage; (3) Biolink validity / QC; (4) efficiency LAST — the prompt states plainly + qualified; (2) coverage; (3) Biolink validity / QC; (4) efficiency LAST; the prompt states plainly that a mappable sheet or evidence column is never sacrificed to save a tool call. - **Digest-first `explode_by`/`split_by` detection**: every previewed table/worksheet ships its injected `column_digest` (separator fractions over the first 500 data rows), and the prompt directs - the agent to read those statistics FIRST — an entity column with a dominant separator gets + the agent to read those statistics FIRST: an entity column with a dominant separator gets `explode_by` for exactly that separator; `read_table` is justified only for rows beyond the digest's scan window. - **ReAct, planning off**: `CodeAgent` is a ReAct loop, but periodic re-planning is disabled (`planning_interval=None`): each planning turn is a whole extra LLM round trip carrying the full - prompt, and the task already prescribes a fixed short workflow (derive → build → answer): on a + prompt, and the task already prescribes a fixed short workflow (derive -> build -> answer): on a coded build error the agent fixes exactly the named field and rebuilds (at most twice) and never - loops on coverage — the supervisor's deterministic improve loop keeps raising coverage after the + loops on coverage; the supervisor's deterministic improve loop keeps raising coverage after the agent finishes. - **Structured / constrained output**: `derive_config` injects the Section JSON schema; a `final_answer_checks=[validate_table_config]` gate means the agent can only terminate with a config whose **every section** is schema-valid (multi-section configs are validated section-by-section). -- **A regex cookbook**: the prompt teaches the actual semantics agents get wrong — Rust-regex +- **A regex cookbook**: the prompt teaches the actual semantics agents get wrong: Rust-regex substitutions (no backreferences, no lookarounds), single-quoted YAML so backslashes stay literal, `regex` vs `remove` vs `exclude_regex`, and that CURIEs come from resolution or `prefix`/`suffix`, never from capture groups. @@ -617,16 +622,16 @@ The agent's `instructions` make the techniques explicit: take literal tokens; `qualified_predicate: biolink:causes` is the one CURIE-taking exception; `species_context_qualifier` stays banned. - **Predicate specificity**: pick the most-specific predicate the derived association class permits, - chosen from the generated legal-predicate table — never a generic default and never a predicate the + chosen from the generated legal-predicate table, never a generic default and never a predicate the class forbids; `predicate_advice` in the audit report names the exact fix when demotion happens. - **Few-shot exemplars**: the tutorial gene~disease section, the ALAMV6 organism~chemical section, a multi-section config (one config, two tables, each section its own source/url), and a **rich exemplar** combining `explode_by: ";"`, a column qualifier, a regex strip, and the paired - `effect_size`/`effect_type` annotations — every exemplar's predicate is a legal, specific choice for + `effect_size`/`effect_type` annotations; every exemplar's predicate is a legal, specific choice for its category pair (guarded by tests). - **Reflexion-style self-critique (supervisor-side)**: the deterministic `propose_config_edit` / `reflexion_improve` reflect on failing rows, - error codes, and unresolved terms, then make a targeted, schema-valid edit — in the supervisor's + error codes, and unresolved terms, then make a targeted, schema-valid edit. In the supervisor's improve loop, never inside the agent's tool surface. - **Error-recovery prompting**: tools return rich coded errors; the prompt directs the agent to read the code + message and fix precisely that field, never repeating an unchanged config. @@ -655,7 +660,7 @@ The harness scores every run on three objectives and optimizes them as a black b - **Cost**: `RunResult.token_usage` + step count (the API is free; tokens are the proxy). - **Reliability**: failed / wrong / redundant tool-call counts from the `ActionStep` logs. -**LLM-as-judge (semantic dimensions only):** a pointwise **0–3** rubric over schema validity, coverage, +**LLM-as-judge (semantic dimensions only):** a pointwise **0-3** rubric over schema validity, coverage, **Biolink validity**, QC pass, predicate/category appropriateness, provenance completeness, efficiency, and tool-call cleanliness, with **position** (both orderings averaged) and **verbosity** bias mitigation. Deterministic metrics gate the rest; the judge only scores what a metric cannot. Without a judge model, an offline @@ -664,14 +669,14 @@ deterministic heuristic is used. **Optimizers:** - **Reflexion**: the simple first-increment retry (`reflexion_improve`). -- **GEPA**: `dspy.GEPA(metric=gepa_metric, candidate_selection_strategy="pareto", …)` optimizes the +- **GEPA**: `dspy.GEPA(metric=gepa_metric, candidate_selection_strategy="pareto", ...)` optimizes the agent's `instructions` + tool `description`s + exemplars as a **black box** from textual feedback (`gepa_metric` returns `dspy.Prediction(score=weighted_quality, feedback="")`). It is system-agnostic, Pareto-native, and needs few rollouts. -**Reporting:** `pareto_frontier(runs)` returns the **non-dominated set** over (quality ↑, cost ↓, -wrong-calls ↓) and its **knee** (best quality per unit cost). +**Reporting:** `pareto_frontier(runs)` returns the **non-dominated set** over (quality ^, cost v, +wrong-calls v) and its **knee** (best quality per unit cost). ### Real-run prompt optimization (`--optimize`) @@ -711,7 +716,7 @@ tablassert agent PMC11708054 --configuration-file ./graph.yaml \ `--max-metric-calls` bounds the GEPA metric budget. `save_optimized_instructions` / `load_optimized_instructions` persist and reload the prompt (a `{instructions, descriptions}` mapping). Without `--instructions-file` the built-in `INSTRUCTIONS` prompt is used. The committed -`examples/agent/optimized_instructions.yaml` is a GEPA **artifact** from an older seed — do not hand-edit +`examples/agent/optimized_instructions.yaml` is a GEPA **artifact** from an older seed: do not hand-edit it; rerun `--optimize` so GEPA starts from the current (detail-first) seed prompt instead. (A real optimization run needs a live model; the offline suite exercises this path via an injectable `gepa_cls` stub.) diff --git a/docs/api/fullmap.md b/docs/api/fullmap.md index b68e93d2..f04edab4 100644 --- a/docs/api/fullmap.md +++ b/docs/api/fullmap.md @@ -177,7 +177,7 @@ from tablassert.biolink import Categories db = Path("/path/to/fullmap/data/fullmap.redb") lf = pl.LazyFrame({"gene": ["TP53", "BRCA1", "EGFR", "KRAS"]}) lf = level_one(lf, "gene") # canonical tokenize, stem, deduplicate, and sort -lf = level_two(lf, "gene") # remove non-word chars → "gene_two" column +lf = level_two(lf, "gene") # remove non-word chars -> "gene_two" column result = resolve(lf=lf, col="gene", db=db, taxon="9606", prioritize=[Categories.GENE], log=False).collect() @@ -198,7 +198,7 @@ print(result.select(["gene", "gene_name", "gene_category"])) Schema-v5 fullmap databases are rejected by the current resolver. Rebuild existing fullmaps to produce schema v6 before using them with this level-one contract. **`level_two` output** (column: `col + "_two"`): -- All non-word characters removed (`\W+` → `""`) from the `level_one` result +- All non-word characters removed (`\W+` -> `""`) from the `level_one` result - Used as fallback when `level_one` produces no match - Preferred for disease names and free text @@ -218,7 +218,7 @@ See [Quality Control](qc.md) for details. Single-shot inspection core behind the `tablassert quick-map` CLI command: resolve raw terms against a fullmap exactly as a build would, one result frame per input term. It runs the same op -chain a build runs per node column (normalization → probe keys → one batched redb fetch → +chain a build runs per node column (normalization -> probe keys -> one batched redb fetch -> filter/rank/dedup), so the returned rows are the rows `build-kg` would emit for a cell holding that term under a `NodeEncoding` with the same settings. The whole input is ONE batched lookup, never a round trip per term. diff --git a/docs/api/lib.md b/docs/api/lib.md index 8deef39c..9c24b8c1 100644 --- a/docs/api/lib.md +++ b/docs/api/lib.md @@ -110,8 +110,8 @@ result: list[dict[str, Any]] = resolve_many( prioritize=[Categories.GENE], ) -# result[0] → {"original_gene": "TP53", "gene": "HGNC:11998", "gene_name": "TP53", ...} -# result[1] → {"original_gene": "BRCA1", "gene": "HGNC:1100", "gene_name": "BRCA1", ...} +# result[0] -> {"original_gene": "TP53", "gene": "HGNC:11998", "gene_name": "TP53", ...} +# result[1] -> {"original_gene": "BRCA1", "gene": "HGNC:1100", "gene_name": "BRCA1", ...} ``` #### Consuming Results @@ -136,7 +136,7 @@ df: pl.DataFrame = pl.DataFrame(result) # Or iterate over resolved rows for row in result: - print(f"{row['gene_name']} → {row['gene']}") + print(f"{row['gene_name']} -> {row['gene']}") ``` ### Comparison With resolve() diff --git a/docs/api/qc.md b/docs/api/qc.md index 16e4a6d4..4a717ecf 100644 --- a/docs/api/qc.md +++ b/docs/api/qc.md @@ -116,7 +116,7 @@ or fuzz.partial_token_sort_ratio(original, preferred) >= 80 _is_abbrev(original, preferred_name) or _is_abbrev(preferred_name, original) ``` -The matcher scans the short form right-to-left against the long form (case-insensitively); the first character of the short form must land on a word boundary of the long form. This rescues the class both fuzzy matching and embedding similarity can miss: `AML` ↔ `acute myeloid leukemia`. +The matcher scans the short form right-to-left against the long form (case-insensitively); the first character of the short form must land on a word boundary of the long form. This rescues the class both fuzzy matching and embedding similarity can miss: `AML` <-> `acute myeloid leukemia`. **Performance:** O(n) character scans per row, no model inference. @@ -181,10 +181,10 @@ validated = fullmap_audit( ``` Input: 1000 rows with entity mappings -Stage 1 (Exact): 700 pass → 300 pending -Stage 2 (Fuzzy): 250 pass → 50 pending -Stage 3 (Abbreviation): 10 pass → 40 pending -Stage 4 (SapBERT): 30 pass → 10 rejected +Stage 1 (Exact): 700 pass -> 300 pending +Stage 2 (Fuzzy): 250 pass -> 50 pending +Stage 3 (Abbreviation): 10 pass -> 40 pending +Stage 4 (SapBERT): 30 pass -> 10 rejected Output: 990 rows (700 + 250 + 10 + 30) ``` @@ -195,7 +195,7 @@ Output: 990 rows (700 + 250 + 10 + 30) |-------|--------|-----------|----------| | 1 | Exact match / rule-based | Highest | Standardized IDs, acronyms, CURIE-like inputs | | 2 | Fuzzy | Medium | Typos, word reordering | -| 3 | Abbreviation expansion | High | Abbreviation ↔ full-name pairs | +| 3 | Abbreviation expansion | High | Abbreviation <-> full-name pairs | | 4 | SapBERT | High | Synonyms, paraphrases | ### Rejection Logging diff --git a/docs/api/utils.md b/docs/api/utils.md index 52a09533..60903ca8 100644 --- a/docs/api/utils.md +++ b/docs/api/utils.md @@ -150,7 +150,7 @@ Before hashing, the record is normalized so that only *meaning* reaches the dige ### Namespace The domain defaults to `"TABLASSERT"`. When `uuid_fields` is declared it becomes the graph's -`rig.source_info.infores_id`, so two graphs asserting the same triple can never mint the same id — +`rig.source_info.infores_id`, so two graphs asserting the same triple can never mint the same id: the uniqueness that full-record hashing provided by accident becomes structural. `uuid_domain` overrides it for graphs that must deliberately share an id space. @@ -160,14 +160,14 @@ Edges deduplicate on their derived id, so an output file can never carry the sam exact repeat collapses; two genuinely different edges deriving one id abort the build with `uuid-fields-not-a-key`, naming the fields that would disambiguate them. -A graph that *expects* such collisions — e.g. one whose `uuid_fields` are the resolved statement, -so two rows with different raw mention spellings resolve to the same CURIE — can instead set +A graph that *expects* such collisions (e.g. one whose `uuid_fields` are the resolved statement, +so two rows with different raw mention spellings resolve to the same CURIE) can instead set `uuid_on_collision: merge` (requires `uuid_fields`). Divergent same-id records are then folded into one edge: list fields are unioned, deduplicated by content, and sorted (so the merged edge is identical regardless of row order), conflicting scalars keep the first record's value, and a build-log summary reports how many records merged and how many scalar conflicts were arbitrated. -Merge mode buffers one full record per unique id until end-of-stream — the memory cost the default -streaming path avoids — which is why it is opt-in. See +Merge mode buffers one full record per unique id until end-of-stream (the memory cost the default +streaming path avoids), which is why it is opt-in. See [Merging collisions instead](../configuration/graph.md#merging-collisions-instead). ### KGX Compliance diff --git a/docs/cli.md b/docs/cli.md index ca21cb5a..291348c3 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -8,13 +8,13 @@ for the live surface. ## Command index -| Command | Use this to… | +| Command | Use this to... | | --- | --- | -| [`agent`](#agent) | Autonomously derive, build, audit, and improve KG configs from PMC articles | +| [`agent`](#agent) | Autonomously derive, build, audit, and improve KG configs from PMC articles (experimental) | | [`build-fullmap`](#build-fullmap) | Build the embedded fullmap redb used for entity resolution | | [`build-kg`](#build-kg) | Build a KGX NDJSON knowledge graph from a YAML configuration | -| [`distill-export`](#distill-export) | Export a recorded distillation NDJSON dataset to an on-disk Hugging Face dataset | -| [`distill-weigh`](#distill-weigh) | Join distillation records to outcomes and prepare LoRA-SFT training rows | +| [`distill-export`](#distill-export) | Export a recorded distillation NDJSON dataset to an on-disk Hugging Face dataset (experimental) | +| [`distill-weigh`](#distill-weigh) | Join distillation records to outcomes and prepare LoRA-SFT training rows (experimental) | | [`quick-map`](#quick-map) | Show what fullmap entity resolution does with one or more terms | | [`validate`](#validate) | Validate a graph or table configuration without executing it | | [`validate-kgx`](#validate-kgx) | Validate built KGX NDJSON against the Biolink Model | @@ -38,8 +38,13 @@ These are flags on the root `tablassert` command, **not** subcommands. ## agent +!!! warning "Experimental" + The agent surface (`tablassert agent`, its flags, and the workspace layout it writes) is + experimental: the API may change without notice or a deprecation cycle. The core pipeline + (`build-kg`, `validate`, `validate-kgx`, `build-fullmap`, `quick-map`) is stable. + Use this to autonomously turn one or more PMC articles into audited, improved KG configs and graphs -(fetch → derive config → build + audit → improve until coverage maps). Requires the `[agent]` extra +(fetch -> derive config -> build + audit -> improve until coverage maps). Requires the `[agent]` extra (`pip install "tablassert[agent]"`); `--optimize` additionally needs the `[optimize]` extra (`pip install "tablassert[optimize]"`, pulls `dspy`). Both are checked after flag validation and before any model is built or article fetched, so a missing extra is reported with its install @@ -96,14 +101,19 @@ tablassert agent PMC11708054 -f ./graph.yaml ## distill-export +!!! warning "Experimental" + The distillation surface (`--distill` recording, `distill-export`, `distill-weigh`, and the + record/outcome NDJSON schemas) is experimental: the API and file formats may change without + notice or a deprecation cycle. + Use this to convert a distillation dataset recorded with [`agent --distill`](#agent) into an on-disk Hugging Face dataset (`save_to_disk`). Requires the `[distill]` extra (`pip install "tablassert[distill]"`, pulls `datasets`). The raw NDJSON already -loads directly in Unsloth Studio and via `datasets.load_dataset("json", ...)` — this export is +loads directly in Unsloth Studio and via `datasets.load_dataset("json", ...)`; this export is only needed for `datasets`-native workflows. Outcome files are skipped: export recognizes `outcomes.ndjson` by its **content** (a first line -whose `record_type` is `outcome`), not its name, and loads only the record files — a directory +whose `record_type` is `outcome`), not its name, and loads only the record files, so a directory holding nothing else exits 2 with a message that says so. A directory with no `*.ndjson` at all still fails with the original empty-directory error. Keep derived training output (e.g. `distill-weigh`'s `train.ndjson`) in a separate directory: export loads every record `*.ndjson` @@ -114,8 +124,8 @@ The corpus is schema-normalized before loading: export reads every record file, seen anywhere in the corpus, and re-emits every row with an explicit `null` for absent keys into one uniform temporary file (outside the input directory) that is what actually gets passed to `load_dataset`. `datasets` otherwise infers its schema from the **first block of the first file -only** and dies with a `CastError` the moment a later file carries a column that block lacked — -which an append-only corpus spanning schema versions (v1 rows without `run_id` next to v2 rows) +only** and dies with a `CastError` the moment a later file carries a column that block lacked, which an +append-only corpus spanning schema versions (v1 rows without `run_id` next to v2 rows) would always trigger. Normalization makes the schema correct by construction, so a v1-only corpus still exports. A column whose type varies across rows (say `"2"` in one line and `2` in another) is **rejected** with exit 2 naming the column and the types: `datasets` would otherwise @@ -132,6 +142,9 @@ tablassert distill-export --distill-dir .tablassert/agent/distill --out ./hf-dat ## distill-weigh +Experimental, like [`distill-export`](#distill-export): the record/outcome schemas and flags may +change without notice. + Join `agent --distill` records to their sibling outcomes, compute deterministic reward weights, and write one flat training row per input record for LoRA/QLoRA supervised fine-tuning. This is data selection, not RLHF: no reward model or online trainer is involved. Keep the output outside the @@ -150,7 +163,7 @@ tablassert distill-export --distill-dir ./training --out ./hf-dataset | `--policy`, `-p` | string (`threshold`, `best-of-n`, or `replication`) | No | `threshold` | Selection policy; an unknown value is rejected at parse time | | `--threshold`, `-t` | float | No | `0.75` | Minimum weight for the `threshold` policy | | `--top-n`, `-tn` | int | No | `2` | Number retained per `pmc_id` group for `best-of-n` | -| `--replication-k`, `-rk` | int | No | `2` | Replication slope for `replication`, bounded to 0–3 | +| `--replication-k`, `-rk` | int | No | `2` | Replication slope for `replication`, bounded to 0-3 | | `--reward-config`, `-rc` | Path | No | `None` | YAML/JSON reward configuration override | | `--edge-ref` | float | No | `None` | Breadth reference override; otherwise the corpus median | | `--purpose` | str | No | `agent` | Keep this purpose, or use literal `all` to disable filtering | @@ -170,7 +183,7 @@ are counts on rows, not physical row duplication. ## build-fullmap Use this to obtain the embedded `fullmap.redb` entity-resolution database. Every database this -command installs is filtered by the built-in **top-100 experimental-taxon allowlist** — there is no +command installs is filtered by the built-in **top-100 experimental-taxon allowlist**; there is no flag for it. By default it first tries to **download a prebuilt database** published for this Tablassert version; `--force` skips that and builds from RENCI BABEL exports instead (download class + synonym files, then build a single redb). @@ -203,15 +216,15 @@ Every one of those invocations produces a database filtered by the checked-in By default `build-fullmap` looks for a prebuilt `fullmap.tar.zst` at `https://stars.renci.org/var/babel_outputs//fullmap//` (the version directory is the **installed Tablassert package version**, never hardcoded), verifies it against the -published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension, streaming zstd → +published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension, streaming zstd -> tar with the GIL released (the decompressed tar never touches disk), then validating the extracted primary + shards against the force-build contract (exact `v6` schema, a recorded `build_id`, the exact shard set, per-shard `build_id` equality, and a `META.taxon_allowlist` identity matching the built-in allowlist) before atomically renaming them into place. If no prebuilt exists for this version (or the download, extraction, or identity check fails), it falls back to a from-scratch BABEL build and logs a -warning — an archive published without the current filter is never installed. +warning: an archive published without the current filter is never installed. -The downloader is chosen automatically — there is no flag to pass: with the optional `[aria2]` extra +The downloader is chosen automatically; there is no flag to pass. With the optional `[aria2]` extra installed (`pip install "tablassert[aria2]"`; Linux/Windows wheels only), both the prebuilt archive and the BABEL files download through the bundled aria2c binary with resumable segmented downloads; without the extra, Tablassert's Python downloader is used. The extra ships no macOS wheels, so a @@ -224,7 +237,7 @@ A database already present at `--output` is reused only when it carries that sam an unfiltered (or differently filtered) leftover from an older Tablassert is rebuilt, with a warning. Pass `--force` to rebuild unconditionally. -A from-scratch build parallelizes automatically across all available CPU threads — on Linux the +A from-scratch build parallelizes automatically across all available CPU threads: on Linux the worker count is capped by available memory (~2 GB per thread, read from `/proc/meminfo`) to avoid OOMs. There is no flag to tune. @@ -234,12 +247,12 @@ See [Fullmap](fullmap.md) for the data pipeline, output schema, and graph-config ## quick-map -Use this to see what fullmap entity resolution will do with one or more terms — the rows a +Use this to see what fullmap entity resolution will do with one or more terms (the rows a `build-kg` run would emit for a cell holding that term under the filters shown. Each term runs -through the exact build path (level-one/level-two normalization → probe-key extraction → redb -fetch → filter, rank, dedup), one `rich` table per term in input order, with the term's normalized +through the exact build path (level-one/level-two normalization -> probe-key extraction -> redb +fetch -> filter, rank, dedup), one `rich` table per term in input order, with the term's normalized probe keys in the title so a miss is diagnosable. The whole input is one batched redb round trip, -never one per term. A term with no matches prints a `no matches` line and still exits 0 — a miss is +never one per term. A term with no matches prints a `no matches` line and still exits 0: a miss is a finding, not a failure; only usage errors and an unreadable fullmap exit 2. ```bash @@ -271,14 +284,14 @@ tablassert quick-map "TNF-alpha" -f ./fullmap -t 9606 -p Gene -a Disease -ep OMI ``` !!! note "The `--prioritize` / `--avoid` vocabulary" - These flags accept the **live Biolink entity category names** — the same vocabulary the - `prioritize` / `avoid` config keys accept — validated at run time rather than enumerated in + These flags accept the **live Biolink entity category names** (the same vocabulary the + `prioritize` / `avoid` config keys accept), validated at run time rather than enumerated in `--help`, because the `Categories` enum is built dynamically from the installed biolink-model. An invalid value exits 2 naming the nearest valid names. !!! note "`-f` means `--fullmap` here" On `build-kg`, `validate`, and `agent`, `-f` is `--configuration-file`; on `build-fullmap` it is - `--force`. Aliases are per-command, and `quick-map` takes no config and has no force mode — + `--force`. Aliases are per-command, and `quick-map` takes no config and has no force mode. `-f` is its one required input path. --- @@ -298,15 +311,15 @@ The positional `GRAPH-CONFIGURATION-FILE` (also `--configuration-file`, `-f`) is | --- | --- | --- | --- | --- | | `GRAPH-CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | n/a | Graph YAML | | `--release`, `-r` | Flag | No | `False` | Emit a slim, significant-only graph (drops `not_significant` edges before resolution) | -| `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact → fuzzy → abbreviation → SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON: no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), no stray whitespace (verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies), and -- when any section declares a `category_override` -- no edge demoted to bare `biolink:Association` (a row escaped its pin and shipped without the class-specific slots `prune_to_class` nulled into `has_supporting_studies`; the assertion is graph-wide, so one pinned section also gates edges from unpinned sections) and fails the build (non-zero exit) on any violation | +| `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact -> fuzzy -> abbreviation -> SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON: no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), no stray whitespace (verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies), and, when any section declares a `category_override`, no edge demoted to bare `biolink:Association` (a row escaped its pin and shipped without the class-specific slots `prune_to_class` nulled into `has_supporting_studies`; the assertion is graph-wide, so one pinned section also gates edges from unpinned sections) and fails the build (non-zero exit) on any violation | | `--log`, `-l` | Flag | No | `False` | Enable verbose per-section logging; requires the `[log]` extra, and without it Tablassert produces no logs | -| `--head`, `-hd` | Flag | No | `False` | Fast output-shape preview: ≤5 random rows/section, cached to `.head.parquet`, never clobbers a full build | +| `--head`, `-hd` | Flag | No | `False` | Fast output-shape preview: <=5 random rows/section, cached to `.head.parquet`, never clobbers a full build | ```bash tablassert build-kg graph.yaml --qc --log ``` -The parallel fullmap reads behind entity resolution are automatic: large lookup batches (≥ 1024 +The parallel fullmap reads behind entity resolution are automatic: large lookup batches (>= 1024 terms) fan out across the record-shard files on all available CPU threads (redb readers share-lock, so they never contend), while smaller batches stay serial. There is no flag to tune, and results are identical at any worker count. @@ -321,8 +334,8 @@ incomplete RIG fails the build with `[rig-validation-failed]` and nothing is emi [Graph Configuration](configuration/graph.md). ??? info "Build progress & stages" - The build runs six parallel stages (Loading Tables → Extracting Sections → Building TCode → - Collecting Instructions → Building Subgraphs → Compiling Graph) plus a seventh, Studying Graph, + The build runs six parallel stages (Loading Tables -> Extracting Sections -> Building TCode -> + Collecting Instructions -> Building Subgraphs -> Compiling Graph) plus a seventh, Studying Graph, only when `--qc` is passed. They run under a three-row live progress block (stage header; section bar with count/elapsed/ETA; in-flight item detail). Each completed stage prints a green `✓ Stage N · NAME · elapsed` line above the live block. During Building @@ -440,4 +453,4 @@ once a model release drops the requirement. - **[Tutorial](tutorial.md)**: complete example walkthrough - **[Configuration Guide](configuration/graph.md)**: YAML configuration reference - **[Fullmap](fullmap.md)**: entity-resolution database build and schema -- **[Agent](agent.md)**: autonomous PMC → KG pipeline depth +- **[Agent](agent.md)**: autonomous PMC -> KG pipeline depth diff --git a/docs/configuration/advanced-example.md b/docs/configuration/advanced-example.md index 38406a15..2efb6a84 100644 --- a/docs/configuration/advanced-example.md +++ b/docs/configuration/advanced-example.md @@ -74,19 +74,19 @@ whole table or `method: column` to pull per-row notes from the source (see - **Excel column letters**: `encoding: A`/`B`/`C` reference the first/second/third columns of the headerless source (organism names, Spearman rho, p-value). -- **Regex pipeline**: the subject runs three substitutions in order: `.*g__` → `` - (`d__Bacteria;p__Firmicutes;g__Lactobacillus` → `Lactobacillus`), then `;s__` → ` ` - (`Lactobacillus;s__rhamnosus` → `Lactobacillus rhamnosus`), then `sp` → `sp. `. +- **Regex pipeline**: the subject runs three substitutions in order: `.*g__` -> `` + (`d__Bacteria;p__Firmicutes;g__Lactobacillus` -> `Lactobacillus`), then `;s__` -> ` ` + (`Lactobacillus;s__rhamnosus` -> `Lactobacillus rhamnosus`), then `sp` -> `sp. `. - **Taxonomic filtering**: `prioritize: [OrganismTaxon]` + `avoid: [Gene]` stop "Lactobacillus" resolving to a similarly-named gene. - **Mixed annotations**: `method: value` for constants (same every row), `method: column` for per-row values. - **Subject-predicate-object**: subject varies per row (column), predicate `correlated_with` is - fixed, object `CHEBI:41774` is fixed → `Lactobacillus rhamnosus --[correlated_with]--> 13C-tamoxifen`. + fixed, object `CHEBI:41774` is fixed -> `Lactobacillus rhamnosus --[correlated_with]--> 13C-tamoxifen`. ??? note "Regex dialect constraint" Each `pattern` is handed to Polars `str.replace_all()` (Rust `regex` crate). **Backreferences - (`\1`, `\2`, …) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(? 0 and rho < 0. ## Dual-Column Mapping Both subject and object come from columns, so both nodes undergo entity resolution. **Use case:** -correlation tables linking two biological entities per row (e.g., metabolite ↔ microbe). +correlation tables linking two biological entities per row (e.g., metabolite <-> microbe). ```yaml template: @@ -203,7 +203,7 @@ template: encoding: B # Column B: microbe names prioritize: [OrganismTaxon] regex: - - {pattern: _, replacement: ' '} # "Lactobacillus_rhamnosus" → "Lactobacillus rhamnosus" + - {pattern: _, replacement: ' '} # "Lactobacillus_rhamnosus" -> "Lactobacillus rhamnosus" provenance: repo: PMC @@ -216,7 +216,7 @@ template: ``` Each column-mapped node gets its own `prioritize` list to guide disambiguation. `remove` strips each -listed pattern (replace with empty string); `regex` applies an ordered `pattern`→`replacement` list; +listed pattern (replace with empty string); `regex` applies an ordered `pattern`->`replacement` list; both transform cell text in place before resolution, and neither drops rows. --- diff --git a/docs/configuration/graph.md b/docs/configuration/graph.md index cc2d5411..8ca3a302 100644 --- a/docs/configuration/graph.md +++ b/docs/configuration/graph.md @@ -34,15 +34,15 @@ The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation | `uuid_fields` | List[str] | Edge fields that constitute edge identity. Only these feed the derived edge `id` (see [Stable edge ids](#stable-edge-ids)) | | `uuid_domain` | String | Explicit UUID namespace. Defaults to `rig.source_info.infores_id` when `uuid_fields` is set, `TABLASSERT` otherwise | | `uuid_on_collision` | `error` \| `merge` | What to do when two different edges derive one id. `error` (default) aborts; `merge` folds them into one edge. Requires `uuid_fields` (see [Merging collisions instead](#merging-collisions-instead)) | -| `qc_similarity_threshold` | Float 0–1 | SapBERT cosine threshold used by QC | -| `qc_fuzzy_ratio_threshold` | Float 0–100 | RapidFuzz ratio threshold used by QC | -| `qc_fuzzy_partial_threshold` | Float 0–100 | RapidFuzz partial/token-set threshold used by QC | +| `qc_similarity_threshold` | Float 0-1 | SapBERT cosine threshold used by QC | +| `qc_fuzzy_ratio_threshold` | Float 0-100 | RapidFuzz ratio threshold used by QC | +| `qc_fuzzy_partial_threshold` | Float 0-100 | RapidFuzz partial/token-set threshold used by QC | | `qc_aliases` | Mapping[String, String] | Source-text aliases applied before QC similarity stages | ## Stable edge ids Every edge gets a deterministic `id`: a UUID v3 derived from the edge itself. By default it is -derived from the **whole record**, which makes it maximally brittle — a corrected `p_value`, a new +derived from the **whole record**, which makes it maximally brittle: a corrected `p_value`, a new `supporting_text` entry, a reordered source row, or a Biolink release that renames a slot all mint a brand-new id. Downstream Translator consumers then see a new edge where they should see the same edge with updated attributes. @@ -59,14 +59,14 @@ Everything else is then free to change without moving the id. Start from what identifies an assertion, and what each entry buys you: -- **`subject` / `predicate` / `object`** — the assertion itself. -- **`publications`** — the evidence it rests on. -- **`has_supporting_studies`** — carries `has_study_results[].id` (`row:`), the **row +- **`subject` / `predicate` / `object`**: the assertion itself. +- **`publications`**: the evidence it rests on. +- **`has_supporting_studies`**: carries `has_study_results[].id` (`row:`), the **row discriminator**. Include it whenever one table contributes several rows that share a subject, predicate, and object. It also carries the `study_*` metadata, but those come from per-section config and are far more stable than `p_value` or `effect_size`. -Add qualifiers (`object_direction_qualifier`, `anatomical_context_qualifier`, …) when they +Add qualifiers (`object_direction_qualifier`, `anatomical_context_qualifier`, ...) when they *distinguish* assertions rather than merely describe them. Leave out anything that is an observation *about* the edge rather than the edge's identity: @@ -74,7 +74,7 @@ Leave out anything that is an observation *about* the edge rather than the edge' `sources`, `category`, `knowledge_level`, `agent_type`. **Then build, and let the failure tell you what is missing.** That list is a starting point, not an -answer — whether it is a key depends on your data, and the only way to find out is to run it. On a +answer: whether it is a key depends on your data, and the only way to find out is to run it. On a real 1.27M-edge graph the set above left 2,476 collisions (0.2% of edges): pairs whose subject, predicate, object, publication and row were identical, differing only in the NLP level recorded in `supporting_text` because two raw strings had resolved onto the same CURIE. Adding @@ -90,8 +90,8 @@ every row) and rebuilding: | | edge ids that changed | |---|---| -| no `uuid_fields` (whole-record hash) | 930,081 of 1,265,355 — **73%** | -| `uuid_fields` declared | 0 of 1,265,355 — **none** | +| no `uuid_fields` (whole-record hash) | 930,081 of 1,265,355 (**73%**) | +| `uuid_fields` declared | 0 of 1,265,355 (**none**) | ### The field set must be a key @@ -109,15 +109,15 @@ Add a discriminating field to `uuid_fields` (...). The fix is whatever the message names: add the qualifier that separates them, `has_supporting_studies` for the source row, or the statistic that genuinely differs. Two rows with -an identical subject, predicate, and object that differ only in `p_value` are exactly this case — -and were previously producing two ids for what config claimed was one assertion. +an identical subject, predicate, and object that differ only in `p_value` are exactly this case, +and were previously producing two ids for what the config claimed was one assertion. An exact duplicate is *not* a violation: identical edges collapse, as they always have. ### Merging collisions instead Sometimes a collision is not a config mistake but the data working as intended: two rows with -different raw mention spellings resolve to the same CURIE, so they derive one id — and the right +different raw mention spellings resolve to the same CURIE, so they derive one id, and the right answer is one edge with the combined evidence, not a failed build. Opt in with: ```yaml @@ -128,22 +128,22 @@ uuid_on_collision: merge Under `merge`, a divergent same-id record is folded into the first record that claimed the id: - **list fields** (`publications`, `sources`, `source_record_urls`, `supporting_text`, `category`, - `upstream_resource_ids`, `has_supporting_studies`, …) are unioned, deduplicated, and **sorted**, + `upstream_resource_ids`, `has_supporting_studies`, ...) are unioned, deduplicated, and **sorted**, so the merged edge is identical regardless of which row arrived first. Object entries such as - `sources[]` dedup by content — key order alone never keeps two copies. + `sources[]` dedup by content; key order alone never keeps two copies. - **scalar fields** keep the lexicographically smallest canonical value; each conflict is counted and reported in a build-log summary. -- fields only the later record carries are copied over — first-wins arbitrates *conflicts*, not +- fields only the later record carries are copied over: first-wins arbitrates *conflicts*, not additions. - **one scalar is exempt from first-wins:** when the merged record carries the build-internal - `supporting_case_ids` list — the case IDs behind a `number_of_cases` count — the merged count + `supporting_case_ids` list (the case IDs behind a `number_of_cases` count), the merged count becomes the length of the unioned list, so a case ID shared by both records counts once where first-wins would under-report and summing would double-count. The superseded divergence is not reported as a scalar conflict, and the carrier list is stripped from every edge before write in both modes, so it never ships in the final NDJSON. `merge` requires `uuid_fields` (under the whole-record hash every field is an identity field, so -two different records can never share an id — there would be nothing to merge) and is rejected +two different records can never share an id; there would be nothing to merge) and is rejected without it as `uuid-merge-without-fields`. The trade-off is memory: merge mode buffers one full record per unique id and writes edges only at @@ -156,8 +156,8 @@ Because a narrow field set no longer distinguishes graphs by accident, declaring moves the UUID namespace onto the graph's own `rig.source_info.infores_id`. Two graphs asserting the same triple from the same publication then still derive different ids, structurally. -Set `uuid_domain` only when graphs must deliberately **share** an id space — a KG compiled in -shards, or one renamed across versions that has to keep its published ids: +Set `uuid_domain` only when graphs must deliberately **share** an id space: a KG compiled in +shards, or one renamed across versions that has to keep its published ids. ```yaml uuid_domain: infores:multiomicskg diff --git a/docs/configuration/table.md b/docs/configuration/table.md index e2b304ae..3dfcec8c 100644 --- a/docs/configuration/table.md +++ b/docs/configuration/table.md @@ -187,7 +187,7 @@ Defines subject-predicate-object relationships. | `predicate` | String | No | Biolink predicate. Defaults to `"related_to"`. Three DAKP prevention predicates are deliberate local extensions ahead of Biolink support (see `biolink.PREDICATE_OVERRIDES`): `prevents`, `applied_to_prevent`, `contraindicated_in_the_prevention_of`. | | `object` | NodeEncoding | Yes | Object entity configuration | | `qualifiers` | List[Qualifier] | No | Edge qualifiers (context) | -| `category_override` | Map[Categories, EdgeCategories] | No | Pin the association class per resolved object category (bare names, no `biolink:` prefix), replacing the derived (subject, object) pair lookup for those rows. Rows whose object category is absent from the map derive as before. Pinned classes are still reconciled against the section predicate (a class whose `predicate` slot rejects it is walked up the association hierarchy, with a warning at config time). Use it when one section mixes object categories the pair lookup merges — e.g. `Disease` and `PhenotypicFeature` both derive to `ChemicalEntityToDiseaseOrPhenotypicFeatureAssociation`, but only `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` receive the class-scoped `regulatory_approvals` grant and declare slots like `number_of_cases`. Rows pinned to either of those classes may additionally carry `disease_context_qualifier` plus DAKP's sparse qualifier stack (`anatomical_context_qualifier`, `sex_qualifier`, `population_context_qualifier`, `frequency_qualifier`, `temporal_context_qualifier`) under deliberate Tablassert policy grants. Deliberately ungranted: `species_context_qualifier` (disabled -- never emittable) plus `temporal_interval_qualifier` and `severity_qualifier` (unsatisfiable -- attached to no class, so a grant could never validate). | +| `category_override` | Map[Categories, EdgeCategories] | No | Pin the association class per resolved object category (bare names, no `biolink:` prefix), replacing the derived (subject, object) pair lookup for those rows. Rows whose object category is absent from the map derive as before. Pinned classes are still reconciled against the section predicate (a class whose `predicate` slot rejects it is walked up the association hierarchy, with a warning at config time). Use it when one section mixes object categories the pair lookup merges: e.g. `Disease` and `PhenotypicFeature` both derive to `ChemicalEntityToDiseaseOrPhenotypicFeatureAssociation`, but only `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` receive the class-scoped `regulatory_approvals` grant and declare slots like `number_of_cases`. Rows pinned to either of those classes may additionally carry `disease_context_qualifier` plus DAKP's sparse qualifier stack (`anatomical_context_qualifier`, `sex_qualifier`, `population_context_qualifier`, `frequency_qualifier`, `temporal_context_qualifier`) under deliberate Tablassert policy grants. Deliberately ungranted: `species_context_qualifier` (disabled -- never emittable) plus `temporal_interval_qualifier` and `severity_qualifier` (unsatisfiable -- attached to no class, so a grant could never validate). | **Example:** ```yaml @@ -345,7 +345,7 @@ subject: Unlike `regex` and `remove`, which rewrite the cell text before resolution, `exclude_prefixes` and `exclude_regex` filter resolved CURIEs after the lookup has run: they shrink the candidate set during entity resolution, and a cell whose candidates are all dropped is left unresolved. What an unresolved cell costs depends on the node it belongs to: an unresolved subject, object, or strict qualifier (`nullable: false`, the default) drops the row, exactly like any other resolution failure, whereas a qualifier declared `nullable: true` keeps the edge and omits the qualifier key for the row (see [Qualifiers](#qualifiers)). -`exclude_prefixes` matches on the namespace prefix — the text before the first `:` of a resolved CURIE (a prefix-less CURIE counts as its own prefix) — and drops every candidate whose prefix is listed. The comparison is a membership test against the listed strings, so matching is **exact and case-sensitive**: `OMIM` drops `OMIM:100100`, but leaves both `OMIMPS:100` (a different prefix, not a prefix match) and `omim:100100` (different case) in the candidate set. `exclude_regex` drops every candidate whose CURIE matches any pattern; matching is likewise case-sensitive and unanchored (Polars `str.contains`), so `^omim:` matches nothing in `OMIM:100100` — use the casing the CURIEs actually carry, and anchor with `^`/`$` to pin a whole CURIE. The Polars/Rust regex dialect constraints documented for `regex` apply to `exclude_regex` too, and an empty or whitespace-only `exclude_regex` entry is rejected at config time with `regex-bad-pattern` because an empty pattern would match every CURIE and silently drop all candidates. +`exclude_prefixes` matches on the namespace prefix (the text before the first `:` of a resolved CURIE; a prefix-less CURIE counts as its own prefix) and drops every candidate whose prefix is listed. The comparison is a membership test against the listed strings, so matching is **exact and case-sensitive**: `OMIM` drops `OMIM:100100`, but leaves both `OMIMPS:100` (a different prefix, not a prefix match) and `omim:100100` (different case) in the candidate set. `exclude_regex` drops every candidate whose CURIE matches any pattern; matching is likewise case-sensitive and unanchored (Polars `str.contains`), so `^omim:` matches nothing in `OMIM:100100`; use the casing the CURIEs actually carry, and anchor with `^`/`$` to pin a whole CURIE. The Polars/Rust regex dialect constraints documented for `regex` apply to `exclude_regex` too, and an empty or whitespace-only `exclude_regex` entry is rejected at config time with `regex-bad-pattern` because an empty pattern would match every CURIE and silently drop all candidates. #### Text Transformations @@ -361,7 +361,7 @@ subject: Executed in order. -> **Regex dialect:** Patterns are passed directly to Polars `str.replace_all()`, which uses the Rust [`regex`](https://docs.rs/regex/) crate. Only features supported by that engine work: in particular, **backreferences (`\1`, `\2`, …) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(? **Regex dialect:** Patterns are passed directly to Polars `str.replace_all()`, which uses the Rust [`regex`](https://docs.rs/regex/) crate. Only features supported by that engine work: in particular, **backreferences (`\1`, `\2`, ...) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(? "CUSTOM:123" ``` **Output columns: `original_` vs `_pre_resolution`** - For every subject/object/qualifier node, the pipeline snapshots the cell value into two columns at different stages: @@ -424,7 +424,7 @@ annotations: object: method: column encoding: B - explode_by: ";" # "P1;P2;P3" → 3 separate edges + explode_by: ";" # "P1;P2;P3" -> 3 separate edges ``` #### Mathematical Transformations @@ -480,7 +480,7 @@ Required metadata about data source. | Field | Type | Required | Description | |-------|------|----------|-------------| | `repo` | String | No | Repository. Defaults to `"PMC"`; allowed values are `"PMC"`, `"PMID"`. | -| `publication` | String | Yes, unless `override` is set | Repository-local identifier emitted as a CURIE. For `repo: PMC` the value **must** start with `PMC` followed by digits (e.g., `"PMC11708054"`, `"PMC123"`) and is emitted under the `PMCID:` namespace as `PMCID:PMC...`; for `repo: PMID` it is emitted as `PMID:` (e.g., `"11708054"` → `PMID:11708054`). | +| `publication` | String | Yes, unless `override` is set | Repository-local identifier emitted as a CURIE. For `repo: PMC` the value **must** start with `PMC` followed by digits (e.g., `"PMC11708054"`, `"PMC123"`) and is emitted under the `PMCID:` namespace as `PMCID:PMC...`; for `repo: PMID` it is emitted as `PMID:` (e.g., `"11708054"` -> `PMID:11708054`). | | `knowledge_level` | String | No | Biolink KL/AT knowledge level of produced edges. Defaults to `"statistical_association"`; allowed values are the `KnowledgeLevels` enum (e.g., `knowledge_assertion`, `logical_entailment`, `prediction`, `statistical_association`, `text_co_occurrence`, `observation`, `not_provided`). | | `agent_type` | String | No | Biolink KL/AT agent type responsible for produced edges. Defaults to `"data_analysis_pipeline"`; allowed values are the `AgentTypes` enum (e.g., `manual_agent`, `automated_agent`, `data_analysis_pipeline`, `computational_model`, `text_mining_agent`, `image_processing_agent`, `manual_validation_of_automated_agent`, `not_provided`). | | `override` | Object | No | Manual provenance for non-PMC/PMID sources. When set, it replaces repo/publication-derived provenance and `publication` must be omitted. | @@ -626,7 +626,7 @@ The supporting study's identity is disjoint by construction: `Study.id` is the s Before the allow-list sweep runs, tablassert renames statistical columns to their canonical Biolink names so source headers do not have to match exactly. Recognition is delimiter-anchored (spaces, `_`, `-`, `.` are interchangeable) and the **best fuzzy match per target wins, with an existing canonical column always preferred** over a higher-scoring spaced alias. -**Losing candidates are dropped, for every target.** A loser is another spelling of the value that already reached the canonical slot, and anything left on the frame would be folded into `supporting_text` — emitting the same measurement twice, once as a typed Biolink slot and once as free text. This has a cost worth knowing: if a table's headers are genuinely *different* statistics that all recognize onto one slot (`beta` beside `odds ratio`, or a two-analysis header set like `p_value_analysis1` / `p_value_analysis2`), **only the winner survives**. Give such columns names the table below does not recognize when every one of them must reach the edge. +**Losing candidates are dropped, for every target.** A loser is another spelling of the value that already reached the canonical slot, and anything left on the frame would be folded into `supporting_text`, emitting the same measurement twice, once as a typed Biolink slot and once as free text. This has a cost worth knowing: if a table's headers are genuinely *different* statistics that all recognize onto one slot (`beta` beside `odds ratio`, or a two-analysis header set like `p_value_analysis1` / `p_value_analysis2`), **only the winner survives**. Give such columns names the table below does not recognize when every one of them must reach the edge. | Recognized as | Canonical name | Typical source spellings | |---|---|---| @@ -637,9 +637,9 @@ Before the allow-list sweep runs, tablassert renames statistical columns to thei | Effect size | `effect_size` | `effect size`, `odds ratio`, `hazard ratio`, `beta`, `log2FC`, `correlation`, `rho` (and the legacy `relationship_strength`) | | Effect type | `effect_type` | `effect type`, `effect metric`, `statistic type`, `metric` | -- **Negative-log p-value columns are un-logged, not shipped verbatim.** A header carrying an explicit negation marker (`negative`, `negated`, `neg`, or a leading `-`) before a `log` / `log10` p-or-q token reports a `-log10` score rather than a probability, so the value is converted back to its original scale (`p = 10 ** -x`) as the column is renamed onto `p_value` / `adjusted_p_value`, and again when the significance band is derived. Left alone, a `-log10(p)` of 8 would emit `p_value` 8.0 and band as `not_significant`, the exact inverse of the truth. The trailing token has to be a *complete* p/q-value token (or a bare delimited `P`), so a word that merely starts with p or q (`negative log protein`) is not matched. A plain `log10 p value` with no negation marker is deliberately left verbatim, because the sign convention is ambiguous there. When a raw p/q column and a `-log10` alias of the same statistic both appear, the raw column always wins selection and the alias is dropped, so a short raw name (`P`, `FDR`) cannot lose to a long alias it would then be un-logged over — and the edge never carries `p_value` 0.03 beside a `"negative log10 p value: 8.0"` supporting-text entry, which reads as a contradiction rather than a duplicate. Scores extreme enough to underflow Float64 are floored at `0.0`, which is indistinguishable from `p ~ 0` in practice and lands in the same band; nulls stay null. -- **`effect_type` values are also coerced.** Each cell is matched case/separator-insensitively against an alias table (e.g. `"OR"` → `odds_ratio`, `"Cohen's d"` → `cohens_d`, `"Spearman"` → `spearmans_rho`), then by `rapidfuzz` fallback against the 25 permissible `EffectTypes` values; anything matching nothing is dropped to `null` rather than carried through (the Biolink range is the enum). -- **`statistical_significance_qualifier` is auto-derived** from the p-value column into five bands: `very_strongly_significant` (p ≤ 0.001), `strongly_significant` (≤ 0.01), `significant` (≤ 0.05), `suggestive` (≤ 0.10), and `not_significant` (> 0.10). The values are bare `StatisticalSignificanceQualifierEnum` tokens. Biolink-model 4.4.4 attaches the slot to `Association` with the enum as its range, so the band rides the edge as a real field with no `biolink:` prefix. The same rigorous selection picks the source column: a raw `p_value` column is preferred, `adjusted_p_value` is the fallback, and the qualifier is omitted entirely when no p-value column is present. +- **Negative-log p-value columns are un-logged, not shipped verbatim.** A header carrying an explicit negation marker (`negative`, `negated`, `neg`, or a leading `-`) before a `log` / `log10` p-or-q token reports a `-log10` score rather than a probability, so the value is converted back to its original scale (`p = 10 ** -x`) as the column is renamed onto `p_value` / `adjusted_p_value`, and again when the significance band is derived. Left alone, a `-log10(p)` of 8 would emit `p_value` 8.0 and band as `not_significant`, the exact inverse of the truth. The trailing token has to be a *complete* p/q-value token (or a bare delimited `P`), so a word that merely starts with p or q (`negative log protein`) is not matched. A plain `log10 p value` with no negation marker is deliberately left verbatim, because the sign convention is ambiguous there. When a raw p/q column and a `-log10` alias of the same statistic both appear, the raw column always wins selection and the alias is dropped, so a short raw name (`P`, `FDR`) cannot lose to a long alias it would then be un-logged over; the edge never carries `p_value` 0.03 beside a `"negative log10 p value: 8.0"` supporting-text entry, which reads as a contradiction rather than a duplicate. Scores extreme enough to underflow Float64 are floored at `0.0`, which is indistinguishable from `p ~ 0` in practice and lands in the same band; nulls stay null. +- **`effect_type` values are also coerced.** Each cell is matched case/separator-insensitively against an alias table (e.g. `"OR"` -> `odds_ratio`, `"Cohen's d"` -> `cohens_d`, `"Spearman"` -> `spearmans_rho`), then by `rapidfuzz` fallback against the 25 permissible `EffectTypes` values; anything matching nothing is dropped to `null` rather than carried through (the Biolink range is the enum). +- **`statistical_significance_qualifier` is auto-derived** from the p-value column into five bands: `very_strongly_significant` (p <= 0.001), `strongly_significant` (<= 0.01), `significant` (<= 0.05), `suggestive` (<= 0.10), and `not_significant` (> 0.10). The values are bare `StatisticalSignificanceQualifierEnum` tokens. Biolink-model 4.4.4 attaches the slot to `Association` with the enum as its range, so the band rides the edge as a real field with no `biolink:` prefix. The same rigorous selection picks the source column: a raw `p_value` column is preferred, `adjusted_p_value` is the fallback, and the qualifier is omitted entirely when no p-value column is present. - **`effect_size` and `effect_type` travel as a pair.** A section declaring one without the other does not fail validation: the unpaired annotation is **dropped** from the section with an `UnpairedEffectAnnotationWarning` naming what was dropped and from where, and the section's edges are kept; neither half carries evidence alone. A bare effect size is uninterpretable (0.85 of *what*, an odds ratio or a Spearman rho?), and Biolink PR #1774 only populates `effect_type` alongside a numeric `effect_size`, so the build nulls an unpaired type anyway. Declare both together to retain the full evidence. Alias spellings count: `odds ratio` and the legacy `relationship_strength` coerce to `effect_size`, so both still need a sibling `effect_type`. Use `method: value` when every row shares one statistic and `method: column` when the table provides it; in a `template` + `sections` config the two lists are concatenated, so a constant `effect_type` declared once on the template pairs with each section's own `effect_size` column. - **Biolink class rules are enforced.** `effect_type` is nulled on every row where `effect_size` is null (and nulled entirely when no `effect_size` column exists); `statistical_significance_qualifier` is only set when `p_value`/`adjusted_p_value` is populated, and null p-values yield a null qualifier. diff --git a/docs/examples.md b/docs/examples.md index 8cad9d1d..3a01845d 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -147,7 +147,7 @@ template: **Key techniques:** -- **Regex pipeline** cleans raw taxonomic strings (`d__Bacteria;p__Firmicutes;g__Lactobacillus` → `Lactobacillus`). Patterns must be Polars `str.replace_all()`-compatible (Rust `regex` engine): no backreferences (`\1`, `\2`, …) or lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(? `Lactobacillus`). Patterns must be Polars `str.replace_all()`-compatible (Rust `regex` engine): no backreferences (`\1`, `\2`, ...) or lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?postings index. -Fullmap is built entirely in-process by Tablassert's own Rust extension: no external tool or install step required by default (this is an in-process redb shard scheme, not the older external DuckDB shards). When the optional `[aria2]` extra is installed (`pip install "tablassert[aria2]"`; Linux/Windows wheels only), both download stages — the prebuilt `fullmap.tar.zst` archive and, for a from-scratch build, the BABEL class and synonym files — use the bundled aria2c binary automatically; without it, Tablassert's Python downloader is used. +Fullmap is built entirely in-process by Tablassert's own Rust extension: no external tool or install step required by default (this is an in-process redb shard scheme, not the older external DuckDB shards). When the optional `[aria2]` extra is installed (`pip install "tablassert[aria2]"`; Linux/Windows wheels only), both download stages (the prebuilt `fullmap.tar.zst` archive and, for a from-scratch build, the BABEL class and synonym files) use the bundled aria2c binary automatically; without it, Tablassert's Python downloader is used. ## Build Command @@ -40,25 +40,25 @@ records its allowlist identity in `META.taxon_allowlist`, and that identity gate - A database already present at `--output` is reused only when its recorded identity matches; otherwise it is rebuilt. -See the [CLI Reference → build-fullmap](cli.md#build-fullmap) for the complete flag table (output path, +See the [CLI Reference -> build-fullmap](cli.md#build-fullmap) for the complete flag table (output path, cache directory, BABEL snapshot version, and the `--force` / `-f` rebuild flag), their defaults, and more examples. By default, `build-fullmap` first downloads a **prebuilt** database published for this Tablassert version, a `fullmap.tar.zst` under `.../fullmap//` (the version directory is the installed package version, never hardcoded), verified against a co-published `sha256sum.txt` and -extracted beside `--output` entirely in the Rust extension: it streams the archive through zstd → tar +extracted beside `--output` entirely in the Rust extension: it streams the archive through zstd -> tar (the multi-GB decompressed tar is never materialized on disk) with the GIL released, extracts into a temp directory on the output's filesystem, and, before renaming anything into place, validates the bundle against the same contract a `--force` build must satisfy: the `meta` schema tag is exactly `tablassert.fullmap.v6`, a `build_id` is recorded, the shard files are exactly the set the primary advertises (no gaps, no extras), every shard's `build_id` equals the primary's, and the primary's `META.taxon_allowlist` identity equals the one the built-in allowlist would record. Only a bundle that -passes is atomically renamed into place (primary → `--output`, shards beside it); any failure raises +passes is atomically renamed into place (primary -> `--output`, shards beside it); any failure raises and the command falls back to the from-scratch build below. If no prebuilt is published for this version it falls back the same way; `--force` / `-f` skips the prebuilt attempt and always builds. When the optional `[aria2]` extra is installed, the bundled aria2c from that extra accelerates -**either** download automatically — the multi-GB prebuilt archive is the ideal aria2 use case — and +**either** download automatically (the multi-GB prebuilt archive is the ideal aria2 use case), and the Python downloader is used otherwise. Two facts matter most when planning a build: @@ -79,7 +79,7 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens 2. **Equivalents index**: parse class files into sorted on-disk runs, then k-way merge them into a memory-mapped index mapping each primary CURIE to its equivalents. 3. **Synonym pass**: a producer/consumer pool streams byte-bounded line-chunks; workers dedup CURIEs, - accumulate normalized-term → (CURIE, source) postings, and spill per-shard runs to disk so peak RAM + accumulate normalized-term -> (CURIE, source) postings, and spill per-shard runs to disk so peak RAM stays flat regardless of input size. 4. **Write**: one redb transaction writes the primary tables, then `shard_count` independent k-way merges write the shard `records` files in parallel (one thread per shard). @@ -98,15 +98,15 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens bar stays clean. The progress detail remains file-level (`aria2c downloading`) rather than byte-level in that mode. - **Equivalents index**: class files parse in parallel into sorted on-disk runs, k-way merged into a - single memory-mapped CURIE→equivalents index; only a compact `(hash, offset)` index lives in RAM, + single memory-mapped CURIE->equivalents index; only a compact `(hash, offset)` index lives in RAM, the string data is mmap'd. - **Synonym pass**: uses **intra-file parallelism**: a small pool of producer threads decompresses/reads the synonym files and pushes byte-bounded line-chunks through a bounded channel, and every worker draws from one shared queue, so the few very large files (protein/smallmolecule/gene/drugchemicalconflated) are processed by **all** workers, not one thread each. Per row, the build collects dimension sets (CURIE prefixes, Biolink categories, sources), - assigns compact integer CURIE IDs via a hash-keyed dedup map (`xxh3_128(curie) → id`), and - accumulates normalized-term → (CURIE, source) postings. Each worker drains its per-CURIE rows and + assigns compact integer CURIE IDs via a hash-keyed dedup map (`xxh3_128(curie) -> id`), and + accumulates normalized-term -> (CURIE, source) postings. Each worker drains its per-CURIE rows and term postings to bounded on-disk spill runs once its buffer fills, the term postings partitioned per-shard at spill time (each `run_s{shard}_{id}.bin` holds only terms with `xxh64(term) & (shards-1)` matching that shard), which keeps peak RAM flat regardless of input size @@ -124,7 +124,7 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens occupy about half as many pages, so current builds write ~50% smaller shard files at the same schema). Because every term's postings already live in its own shard's runs, each merge groups a term completely with no cross-shard coordination; as the merge+insert is the bottleneck, - `shard_count` writers deliver ~N× single-threaded write throughput. + `shard_count` writers deliver ~Nx single-threaded write throughput. ### Build Tunables (environment) @@ -134,30 +134,30 @@ override only when targeting an unusual machine. | Variable | Default | Description | |----------|---------|-------------| | `TABLASSERT_FULLMAP_EXCLUDE_PREFIXES` | *(empty)* | Comma-separated CURIE prefixes to drop at build time (e.g. `Publication`). Excluding prefixes you never resolve dramatically cuts build time, peak memory, and database size. InChIKey terms are retained unless excluded explicitly. | -| `TABLASSERT_FULLMAP_CHUNK_BYTES` | `8388608` (8 MiB) | Byte budget per producer→worker line-chunk. Bounded by bytes (not line count) so chunk memory is fixed even for large synonym records. | +| `TABLASSERT_FULLMAP_CHUNK_BYTES` | `8388608` (8 MiB) | Byte budget per producer->worker line-chunk. Bounded by bytes (not line count) so chunk memory is fixed even for large synonym records. | | `TABLASSERT_FULLMAP_PRODUCERS` | `clamp(workers/4, 4, #files)` | Number of producer (decompressor) threads. Decompression far outpaces parallel processing, so a handful keeps all workers fed. | -| `TABLASSERT_FULLMAP_LOCAL_SPILL_ENTRIES` | `1000000` | Per-worker term-posting buffer size before spilling a sorted run to disk. Lower → less RAM, more run files. | -| `TABLASSERT_FULLMAP_CURIE_SPILL_ENTRIES` | `250000` | Per-worker CURIE-row buffer size before spilling to disk. Lower → less RAM, more run files. | +| `TABLASSERT_FULLMAP_LOCAL_SPILL_ENTRIES` | `1000000` | Per-worker term-posting buffer size before spilling a sorted run to disk. Lower -> less RAM, more run files. | +| `TABLASSERT_FULLMAP_CURIE_SPILL_ENTRIES` | `250000` | Per-worker CURIE-row buffer size before spilling to disk. Lower -> less RAM, more run files. | | `TABLASSERT_FULLMAP_EQUIV_SPILL_ENTRIES` | `2000000` | Per-thread equivalents buffer size before spilling during the equivalents-index build. | -| `TABLASSERT_FULLMAP_INSERT_BATCH` | `2000000` | Records buffered per hash-sorted batch during the redb write. Larger → faster writes, modestly more RAM. | +| `TABLASSERT_FULLMAP_INSERT_BATCH` | `2000000` | Records buffered per hash-sorted batch during the redb write. Larger -> faster writes, modestly more RAM. | | `TABLASSERT_FULLMAP_REDB_CACHE_BYTES` | `2147483648` (2 GiB) | redb write-cache size. | | `TABLASSERT_FULLMAP_SPILL_DIR` | `.spill.d` | Directory for intermediate spill runs (removed on success). | ## Output Artifact A primary redb file (default `./fullmap/data/fullmap.redb`) plus its sibling RECORDS shard files -(`fullmap.s0.redb` … `fullmap.s15.redb`, one per shard, named after the output +(`fullmap.s0.redb` ... `fullmap.s15.redb`, one per shard, named after the output file stem in the same directory). The shard count is fixed at 16 (the read path still honors the count recorded in an existing database's `meta` table). Together they hold six tables (see `rust/src/fullmap.rs`): | Table | Description | |-------|-------------| -| `records` | Normalized term (xxhash `u64`) → serialized list of resolution postings (CURIE id, source id). Hash-partitioned across the shard files (`fullmap.s0..s.redb`) by `xxh64(term) & (shards-1)`; a term lives in exactly one shard. | -| `prefixes` | Compact `u16` id → CURIE prefix string (primary file) | -| `categories` | Compact `u16` id → Biolink category string (primary file) | -| `sources` | Compact `u8` id → source metadata (name/version) (primary file) | -| `curies` | Compact `u32` id → CURIE record (CURIE, preferred name, category, taxon, source) (primary file) | +| `records` | Normalized term (xxhash `u64`) -> serialized list of resolution postings (CURIE id, source id). Hash-partitioned across the shard files (`fullmap.s0..s.redb`) by `xxh64(term) & (shards-1)`; a term lives in exactly one shard. | +| `prefixes` | Compact `u16` id -> CURIE prefix string (primary file) | +| `categories` | Compact `u16` id -> Biolink category string (primary file) | +| `sources` | Compact `u8` id -> source metadata (name/version) (primary file) | +| `curies` | Compact `u32` id -> CURIE record (CURIE, preferred name, category, taxon, source) (primary file) | | `meta` | Schema version tag (`tablassert.fullmap.v6`), the shard count (`shards`), the BABEL `source_version` used to build the file, and the allowlist identity (`taxon_allowlist`) recorded for every `build-fullmap` output; absent only in databases built unfiltered through the lower-level surfaces (the Rust API, or `build_fullmap_pipeline`'s `taxon_allowlist=None` default) (primary file) | The shard files must remain alongside the primary file: lookups discover them as siblings of the @@ -168,10 +168,10 @@ Lookups (`lookup_fullmap_terms`) check the primary's `meta` schema tag before re parallel (releasing the GIL, one reader per non-empty shard, re-merged into input order); a mismatched or missing tag raises rather than silently reading incompatible data. Databases built under the older `v1`/`v2`/`v3`/`v4`/`v5` schemas are rejected: there is no automatic schema migration, so a schema bump -(including the v3→v4 move to sharded files, the v4→v5 move to the redb 4 engine, and the v5→v6 move to normalized level-one keys) requires rebuilding +(including the v3->v4 move to sharded files, the v4->v5 move to the redb 4 engine, and the v5->v6 move to normalized level-one keys) requires rebuilding via `tablassert build-fullmap`. -Readers open every fullmap file READ-ONLY with a SHARED file lock (redb ≥ 3 `ReadOnlyDatabase`), so any +Readers open every fullmap file READ-ONLY with a SHARED file lock (redb >= 3 `ReadOnlyDatabase`), so any number of processes can run lookups against the same fullmap concurrently; only a `build-fullmap` rebuild (an exclusive-lock writer) briefly blocks readers. Each lookup pins one primary-plus-shards file generation, cached handles are validated against the file's `(dev, ino)` on every use, so a reader diff --git a/docs/index.md b/docs/index.md index 204f0682..fa58a6b7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,6 +7,20 @@ graphs, declaratively, with entity resolution built in and optional quality cont [Tutorial](tutorial.md). **Automating?** Use the [CLI](cli.md) or the autonomous [Agent](agent.md). +## Who this is for + +Translator ingest authors and knowledge-graph data engineers holding tabular biomedical sources: +association tables, assay results, curated gene-disease lists. Tablassert turns those rows into +KGX nodes and edges from a declarative YAML mapping, resolving free text to standard CURIEs through +an embedded database built from RENCI BABEL exports, so a build needs no hosted resolution service +and no per-source Python transform. It also emits the Resource Ingest Guide (RIG) metadata that +[`NCATSTranslator/translator-ingests`](https://github.com/NCATSTranslator/translator-ingests) +submissions require. + +**Questions or bugs?** Open an issue at +[github.com/SkyeAv/Tablassert/issues](https://github.com/SkyeAv/Tablassert/issues); see +[CONTRIBUTING.md](https://github.com/SkyeAv/Tablassert/blob/main/CONTRIBUTING.md) to contribute. + ## Documentation Sections **Start** @@ -21,6 +35,7 @@ graphs, declaratively, with entity resolution built in and optional quality cont `validate-kgx`, `quick-map`, `agent`, `distill-export`, `distill-weigh` - **[Fullmap](fullmap.md)**: obtain and build the entity-resolution database - **[Agent](agent.md)**: run the autonomous PMC-to-graph pipeline, with checkpoints and tooling + (experimental: API may change) **Reference** diff --git a/docs/installation.md b/docs/installation.md index 30845d22..4671de4f 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,6 +1,6 @@ # Installation -Install Tablassert's Python API, then add the `[cli]` extra to use the `tablassert` command and any of the `rt` / `aria2` / `qc` / `agent` / `optimize` / `distill` / `log` extras that match how you will use it (runtime compatibility, accelerated fullmap downloads, auditing mappings, running the autonomous agent, GEPA prompt optimization, distillation dataset export, or loguru-backed logging). +Install Tablassert's Python API, then add the `[cli]` extra to use the `tablassert` command and any of the `rt` / `aria2` / `qc` / `agent` / `optimize` / `distill` / `log` extras that match how you will use it (runtime compatibility, accelerated fullmap downloads, auditing mappings, running the autonomous agent, GEPA prompt optimization, distillation dataset export, or loguru-backed logging). The `agent`, `optimize`, and `distill` extras back experimental surfaces: their APIs may change without notice. ## Prerequisites @@ -21,40 +21,12 @@ pip install uv ## Installation Methods -### Method 1: Development Installation with UV (Recommended for contributors) - -Best for development, testing, and active work on Tablassert. - -```bash -# Clone the repository -git clone https://github.com/SkyeAv/Tablassert.git -cd Tablassert - -# Install development dependencies, CLI runtime, and optional QC/logging runtimes -uv sync --group dev --extra cli --extra qc --extra log - -# Build the editable Rust extension into the uv environment -uv run maturin develop --manifest-path rust/Cargo.toml - -# Verify the CLI -uv run tablassert --help -``` - -This creates a virtual environment in `.venv/`, installs the development dependencies, and builds the local PyO3 extension. The `tablassert` command is available through `uv run`. See [Development](development.md) and the repository `CONTRIBUTING.md` for the daily edit/check loop. - -### Method 2: Install from PyPI +### Method 1: Install from PyPI Recommended for most users. The base install exposes Tablassert's Python API; add `[cli]` to install the `tablassert` command. QC and other extras are opt-in. ```bash -# Option A: Install the command-line tool with UV -uv tool install "tablassert[cli]" - -# Option B: Install the Python API with pip -pip install tablassert - -# Add the command-line interface to a pip install -pip install "tablassert[cli]" +uv tool install "tablassert[cli]" # or: pip install "tablassert[cli]" ``` #### Optional Extras @@ -64,36 +36,15 @@ pip install "tablassert[cli]" | `cli` | `tablassert` command and rich terminal progress | `cyclopts`, `rich` | | `rt` | Runtime-compatible Polars build | `polars[rtcompat]` | | `aria2` | Bundled aria2c downloader, used automatically by `build-fullmap` when installed (Linux/Windows wheels only) | `aria2==0.0.1b0` (imports as `aria2c`, bundles aria2c) | -| `qc` | QC runtime (exact → fuzzy → abbreviation → SapBERT audit) | `scikit-learn`, `sentence-transformers` (`torch` + `numpy` arrive transitively; `rapidfuzz` is a core dependency) | -| `agent` | Autonomous PMC → KG agent (`tablassert agent`) | `smolagents`, `litellm` | -| `optimize` | GEPA prompt optimization (`tablassert agent --optimize`) | `dspy` | -| `distill` | Distillation dataset export (`tablassert distill-export` → on-disk Hugging Face dataset) | `datasets>=3.0.0` | +| `qc` | QC runtime (exact -> fuzzy -> abbreviation -> SapBERT audit) | `scikit-learn`, `sentence-transformers` (`torch` + `numpy` arrive transitively; `rapidfuzz` is a core dependency) | +| `agent` | Autonomous PMC -> KG agent (`tablassert agent`); experimental, API may change | `smolagents`, `litellm` | +| `optimize` | GEPA prompt optimization (`tablassert agent --optimize`); experimental, API may change | `dspy` | +| `distill` | Distillation dataset export (`tablassert distill-export` -> on-disk Hugging Face dataset); experimental, API may change | `datasets>=3.0.0` | | `log` | loguru-backed file/progress logging (rotation, enqueue) | `loguru` | -```bash -# Install the command-line interface -uv tool install "tablassert[cli]" -pip install "tablassert[cli]" - -# Install with runtime-compatible Polars -# (for CPUs without the required Polars instructions) -uv tool install "tablassert[rt]" - -# pip equivalents -pip install "tablassert[rt]" - -# Install the bundled aria2c downloader -uv tool install "tablassert[aria2]" -pip install "tablassert[aria2]" - -# Install the QC runtime -uv tool install "tablassert[qc]" -pip install "tablassert[qc]" - -# Install the autonomous agent -uv tool install "tablassert[agent]" -pip install "tablassert[agent]" -``` +Install any extra the same way: `uv tool install "tablassert[]"` or +`pip install "tablassert[]"`, and combine them as `"tablassert[cli,qc]"`. Wheels ship for +Linux and macOS; elsewhere pip builds from the sdist, which needs a Rust toolchain. !!! note "`[aria2]` platform and license notes" The `[aria2]` extra depends on the PyPI `aria2` package, which imports as `aria2c` and bundles a static aria2c binary. Its wheels are available for Linux and Windows only; the extra ships no macOS wheels, so a normal macOS install resolves to Tablassert's Python downloader. @@ -126,7 +77,7 @@ fails, not a failure report): | `tablassert agent` | `[agent]` | After flag validation, before any model is built or any article fetched | | `tablassert agent --optimize` | `[agent]` + `[optimize]` | Same point; both are reported at once | | `tablassert distill-export` | `[distill]` | After the recorded-NDJSON input check (an empty `--distill-dir` is reported first, since that typo is the faster loop to close) and before `datasets` is imported | -| `build-fullmap` | `[aria2]` | Before the first download, to pick the downloader — bundled aria2c when the extra is installed, Python downloader otherwise (announced on stderr and logged either way; never a missing-extra failure) | +| `build-fullmap` | `[aria2]` | Before the first download, to pick the downloader: bundled aria2c when the extra is installed, Python downloader otherwise (announced on stderr and logged either way; never a missing-extra failure) | A partially installed extra names every package it is still missing, so installing them is one step rather than a retry loop. Library calls that reach an optional import directly (for example @@ -145,28 +96,32 @@ no logs: it does not create `.tablassert/log/`, write `tablassert.log`, forward progress display, honor `build-kg --log`, or warn about the missing extra. Install `pip install "tablassert[log]"` for loguru-backed file and progress logging (rotation, enqueue). -### Method 3: Install from GitHub main +### Method 2: Install from GitHub main Use this when you want the latest main-branch build. ```bash -# Install from main branch uv tool install "tablassert[cli] @ git+https://github.com/SkyeAv/Tablassert.git@main" ``` -### Method 4: Install from local source +### Method 3: Development install from source (contributors) -For contributors testing local repository changes. +Editable install plus the development tools; needs `uv` and a Rust toolchain. ```bash -# Clone the repository git clone https://github.com/SkyeAv/Tablassert.git cd Tablassert - -# Install Tablassert CLI tool from local source -uv tool install ".[cli]" +uv sync --group dev --extra cli --extra qc --extra log +uv run maturin develop --manifest-path rust/Cargo.toml +uv run tablassert --help ``` +This creates `.venv/`, installs the development dependencies, and builds the local PyO3 extension; +the `tablassert` command is available through `uv run`. To install the CLI as a tool from a local +checkout instead, use `uv tool install ".[cli]"`. See [Development](development.md) and +[`CONTRIBUTING.md`](https://github.com/SkyeAv/Tablassert/blob/main/CONTRIBUTING.md) for the daily +edit/check loop. + ## Verifying Installation Confirm the CLI is on your path (use `uv run tablassert --help` for the in-repo dev environment): @@ -179,33 +134,22 @@ You should see the Tablassert CLI help message with available commands. ## Development Setup -For contributing to Tablassert, use the source install above, then run the local task runner: +For a source checkout, use Method 3 above, then the task runner: ```bash make setup make check ``` -The underlying stable gates are `ruff check` / `ruff format --check`, `pyright`, `pytest`, and `cargo fmt --check` / `cargo test` / `cargo clippy --all-targets -- -D warnings` (see [Development](development.md)). - -Install pre-commit hooks to run the fast lint/format checks automatically before commits (the full gates run in CI): - ```bash uv run pre-commit install ``` -## Upgrading Development Installation - -To upgrade to the latest version: - -```bash -# Pull latest changes -git pull origin main - -# Update dependencies and rebuild the editable extension -uv sync --group dev --extra cli --extra qc --extra log -uv run maturin develop --manifest-path rust/Cargo.toml -``` +`make setup` syncs dependencies and builds the editable extension; `make check` runs the whole gate; +the hook install wires the fast lint/format hooks for commit and push. Gate contents, the daily loop, +and the docs build live in [Development](development.md) and +[`CONTRIBUTING.md`](https://github.com/SkyeAv/Tablassert/blob/main/CONTRIBUTING.md). To pick up +`main`, re-run Method 3's `uv sync` and `maturin develop`. ## Troubleshooting diff --git a/docs/tutorial.md b/docs/tutorial.md index 3ffe2635..d2b3143d 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -8,12 +8,22 @@ configurations, running Tablassert, and examining the output. The input files ship under `docs/examples/` and are validated against the live schema by the test suite; the configurations below reproduce them exactly. -**Time:** 5-10 minutes +**Time:** 5-10 minutes once the fullmap database is present. Obtaining it is a separate, larger +step: `tablassert build-fullmap` downloads a multi-GB prebuilt archive when one is published for +your Tablassert version, and otherwise downloads RENCI BABEL exports and builds the database from +scratch (see [Fullmap](fullmap.md) for sizes, disk requirements, and build tunables). ## Prerequisites -- Tablassert installed (see [Installation](installation.md)) -- Required database: fullmap +- Tablassert installed with the `[cli]` extra (see [Installation](installation.md)) +- The fullmap entity-resolution database. Build or download it once: + + ```bash + tablassert build-fullmap --output ./fullmap/data/fullmap.redb + ``` + + Then point the graph config's `fullmap:` field at `./fullmap` in Step 3. See + [Fullmap](fullmap.md) for what the build produces and how the path is resolved. - Basic familiarity with YAML ## The Data @@ -118,8 +128,10 @@ rig: artifact_base_path: ./tutorial-output ``` -**Important:** Replace `fullmap` with the path to your fullmap redb (and adjust `tables` if your table -config lives elsewhere). The `rig:` section is required; see the +**Important:** Replace `fullmap` with the path to your fullmap redb file or its base directory (the +`tablassert build-fullmap --output ./fullmap/data/fullmap.redb` invocation in +[Prerequisites](#prerequisites) makes `./fullmap` correct), and adjust `tables` if your table +config lives elsewhere. The `rig:` section is required; see the [Graph configuration reference](configuration/graph.md) for every field. **What this does:** @@ -146,7 +158,7 @@ tablassert build-kg tutorial-graph.yaml --qc --log - Reads the source file from disk - Applies row slicing - Resolves entities (genes and diseases) - - Validates with the QC pipeline when `--qc` is passed (exact → fuzzy → abbreviation → SapBERT) + - Validates with the QC pipeline when `--qc` is passed (exact -> fuzzy -> abbreviation -> SapBERT) - Creates subgraph parquet file 3. Aggregates all subgraphs 4. Exports NDJSON files and the RIG @@ -190,7 +202,7 @@ The Resource Ingest Guide records the graph's source metadata and terms of use ( ## Understanding the Transformation -Entity resolution maps text to CURIEs ("TP53" → `HGNC:11998` Gene, "lung cancer" → `MONDO:0008903` Disease); when `--qc` is passed, the QC pipeline validates each mapping across four stages (exact → fuzzy → abbreviation → SapBERT). The output is KGX-compliant nodes and edges with standardized CURIEs, Biolink categories and predicates, provenance, and edge annotations. +Entity resolution maps text to CURIEs ("TP53" -> `HGNC:11998` Gene, "lung cancer" -> `MONDO:0008903` Disease); when `--qc` is passed, the QC pipeline validates each mapping across four stages (exact -> fuzzy -> abbreviation -> SapBERT). The output is KGX-compliant nodes and edges with standardized CURIEs, Biolink categories and predicates, provenance, and edge annotations. ## Next Steps diff --git a/examples/agent/README.md b/examples/agent/README.md index 459ad65d..c44ac3da 100644 --- a/examples/agent/README.md +++ b/examples/agent/README.md @@ -1,5 +1,9 @@ # Agent prompt-optimization artifacts +Experimental: these artifacts belong to the experimental `[agent]` surface +(`--optimize`, `--distill`, the QC scripts). Their formats, flags, and paths may change without +notice or a deprecation cycle. + These artifacts come from running the Tablassert `[agent]` GEPA prompt-optimization path (`tablassert agent --optimize`) against a Qwen OpenAI-compatible endpoint. @@ -58,7 +62,7 @@ library default (the coverage-scoring builds stay serialized on the process-wide ## QC state-directory requirement Configs generated from `gepa-dataset.yaml` point `source.local` at tables under the **GEPA run's state -dir** (here `.tablassert/gepa/downloads/…`, since the dataset's `workdir` is `.tablassert/gepa`), but +dir** (here `.tablassert/gepa/downloads/...`, since the dataset's `workdir` is `.tablassert/gepa`), but `qc/qc_reviewer.py` only reads a config's `source.local` when it resolves INSIDE its own `STATE_DIR/downloads` allowlist (an injection defense, see `get_table_summary`). So the QC scripts must be pointed at the SAME state dir that holds `downloads/`, or the tables must be staged there: diff --git a/llms.txt b/llms.txt index 7f1ba588..188b3a35 100644 --- a/llms.txt +++ b/llms.txt @@ -9,7 +9,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu ## Quickstart - [README](README.md): high-level overview, install snippets, and one-command graph build. - [Documentation Site Home](docs/index.md): docs landing page and navigation context. -- [Installation Guide](docs/installation.md): local/dev/tool installation paths. Optional extras: `cli` (Cyclopts and Rich command-line runtime), `rt` (runtime-compatible Polars), `aria2` (bundled aria2c downloader), `qc` (quality-control runtime), `agent` (autonomous agent), `optimize` (GEPA prompt optimization), `distill` (distillation dataset export), and `log` (loguru-backed logging). +- [Installation Guide](docs/installation.md): three install paths (PyPI, GitHub main, development from source), platform and Rust-toolchain notes, and the missing-extra behavior. Optional extras: `cli`, `rt`, `aria2`, `qc`, `agent`, `optimize`, `distill`, and `log`; the last three back experimental surfaces whose APIs may change. - [CLI Reference](docs/cli.md): the eight commands (`agent`, `build-fullmap`, `build-kg`, `distill-export`, `distill-weigh`, `quick-map`, `validate`, and `validate-kgx`) plus the app-level `--version` flag. - [Tutorial](docs/tutorial.md): first end-to-end run from CSV input to KGX NDJSON output. @@ -20,7 +20,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu - [Tutorial Table Example](docs/examples/tutorial-table.yaml): minimal table configuration. - [Tutorial Graph Example](docs/examples/tutorial-graph.yaml): minimal graph configuration. - [Advanced Example](docs/configuration/advanced-example.md): complex regex, filtering, annotations, and section overrides. -- [Agent Guide](docs/agent.md): autonomous article-to-graph pipeline, checkpoints, and tooling. +- [Agent Guide](docs/agent.md): autonomous article-to-graph pipeline, checkpoints, and tooling. Experimental surface: `agent`, `--optimize`, `--distill`, and the `distill-export` / `distill-weigh` commands may change their API without notice. ## CLI and Runtime - [Fullmap Guide](docs/fullmap.md): entity-resolution database build pipeline and redb schema.