Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/adr/0053-issue-layout-and-entry-facts.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,9 @@ Amended 2026-11-10, after measuring the export of that day (agent's proposal):
- *Coverage:* abbreviations ("UC", "PsA") never match MeSH, so a release that writes the disease only that way gives no indication. Company pipeline pages (decision 5) remain the route to nearly every entry.
- **The reading job (`catalyst indicate-catalysts`, 2026-11-12):** for each published catalyst with one drug tested alone and no indication, plain code takes the source claim that states the event and its stored copy, and both readers run as above. A found indication is kept as a claim: its quote is the sentence, its value the MeSH identifier, the MeSH name, and the sentence's own words; its field is `indication:` and the catalyst's id, so two catalysts never share a claim row. The second reader's pass is a claim check beside it. Every try, found or refused, is a row of `indication_attempt`, so a re-run never pays to ask again about the same catalyst, claim, prompts, and models. A copy that the strict and browser-like readers read differently is refused without a model call (ADR 0044). The job asks nothing unless `indication_gate_problems` is empty, and a role at its cap stops it. It never writes a catalyst; the writer's pass does, in the next pull request. The writer reads indications only through `indication_attempt`, never by claim field, and uses only each catalyst's latest attempt (by `attempted_at`): a later refusal, an earlier run's disease redrawn included, overrides an earlier find (Opus review of PR 127). A copy refused because the two page readers differ is refused without a model call; it comes back only when a prompt or model version changes.
- **Its own gate:** a catalyst whose indication fails these checks publishes as it does today, without an indication. Only the indication waits in the review queue with its reason. No existing check is loosened. This touches the publish gate: test first, and Opus review.
- **Three pull requests:** the MeSH file and matcher (merged, PR 123); the two readers and their checks (PR 124); then the writer, with Opus review.
- **The writer's pass (`catalyst.publish.indication`, 2026-11-12):** plain code under the writer's lock. For each catalyst with no indication, it takes the latest `indication_attempt`; only when that try found one, it checks that the claim is the catalyst's own (`indication:<catalyst id>`), that the second reader's pass is stored, and that the quote is still at its place in the stored copy for the current text version (`gate.match_quote`). Then it sets `indication_id` to the MeSH disease's `indication` row (one per disease, no line or biomarker yet) and writes a change row citing the claim, so the read API shows the sentence among the catalyst's sources. An indication already set is never changed. What is printed is the claim's own words (`IndicationView.name`); the MeSH name stays in the `indication` row for search.
- **When it runs:** the weekday extract run asks both readers about up to its document limit of newly published catalysts right after the writer, then runs the indication pass; `catalyst publish-claims` runs the pass too. No workflow, role, or cap changes: the readers spend under the extractor's and verifier's existing daily caps.
- **Pull requests:** the MeSH file and matcher (PR 123); the two readers and their checks (PR 124); the floors (PR 126); the reading job (PR 127); the writer's pass (this one), each with a separate Opus review.

### Order

Expand Down
5 changes: 3 additions & 2 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,8 @@ Claude Code: tick items (`[x]`) in the same pull request that finishes them.
- [ ] Facts on every entry (owner request 2026-11-10: indication, phase including preclinical, cohort size, named conferences' dates, pipeline and program links; the methods below and the wait for the first issue are the agent's proposals; ADR 0053, decisions 2 to 6), each its own pull request:
- [ ] Indication (ADR 0053, decision 2 as amended): its own quote naming the drug and the disease, matched to one MeSH disease by plain code; gated on its own, so the catalyst still publishes without it; publish gate, test first, Opus review
- [x] MeSH file and matcher: `catalyst collect --sources mesh` stores NLM's yearly descriptor file with its hash; `catalyst.mesh` matches whole words to exactly one disease, refusing narrower, negated, broad, and short-form matches (41% of 371 extracted indications; two samples of 40 all correct)
- [x] The indication's own sentence, read by one model and checked by a second (owner decision 2026-11-10; ADR 0053, decision 2 as amended): plain code offers the visible sentences naming the drug; the first reader (extractor role) picks one and copies the disease; plain code checks it is written there and names one MeSH disease; the second reader (verifier role) confirms the sentence states the drug is aimed at it for this event; scored on 20 gold events and 37 planted cases; three earlier routes not merged (their results are in the ADR)
- [ ] The writer: the indication and its change row, with its own review reason
- [x] The indication's own sentence, read by one model and checked by a second (owner decision 2026-11-10; ADR 0053, decision 2 as amended): plain code offers the visible sentences naming the drug; the first reader (extractor role) picks one and copies the disease; plain code checks it is written there and names one MeSH disease; the second reader (verifier role) confirms the sentence states the drug is aimed at it for this event; scored on 20 gold events and 39 planted cases; three earlier routes not merged (their results are in the ADR)
- [x] The writer: the indication and its change row, with its own review reason (ADR 0053, decision 2): floors as owner paths (PR 126); `catalyst indicate-catalysts` keeps every try in `indication_attempt` with its reason (PR 127); the writer's indication pass sets `indication_id` from each catalyst's latest confirmed try after checking its claim, the second reader's pass, and its quote, with a change row citing the claim; the read API prints the sentence's own words; the weekday extract run asks and writes after publishing. Refusal reasons are in `indication_attempt` and the run's summary; the review-queue page shows them when it is built
- [ ] Conferences: `config/conferences.toml` lists which meetings to follow and their official pages; each page is stored with its hash and its dates and city are quoted claims through the publish gate; plain code adds them when a quote names a followed meeting; an unnamed one stays as written
- [ ] Development stage, preclinical included: a new prompt version with a whole-word check, cross-checked against the registry phase, gold set scored again; adds 🟧 Preclinical to the key
- [ ] Pipeline and program pages: found on each company's own domain, stored with their hash, linked on a "Company pages:" line apart from the sources (they back no value), checked weekly
Expand Down Expand Up @@ -205,6 +205,7 @@ Newest first. Every ADR in `docs/adr` adds a line here.

| Date | Decision |
| --- | --- |
| 2026-11-12 | ADR 0053, decision 2: the writer's indication pass sets a catalyst's `indication_id` only from its latest `indication_attempt` when that try found one, after checking the claim is the catalyst's own, the second reader's pass is stored, and the quote still matches its stored copy; a change row cites the claim; the read API prints the sentence's words, not the MeSH name; the weekday extract run asks and writes right after publishing, inside the existing caps |
| 2026-11-12 | ADR 0053, decision 2: the indication reading job (`catalyst indicate-catalysts`) asks both readers about published catalysts with one drug tested alone and no indication, only while the indication gate is clear; a found indication is a claim (field `indication:<catalyst id>`, quote its sentence) with the second reader's pass as a check; every try is kept in `indication_attempt` so none is paid for twice; it never writes a catalyst; plain code also refuses a disease written straight before another MeSH disease term |
| 2026-11-12 | ADR 0053, decision 2: floors for the indication readers, at the third run's scores (found_right ≥ 0.133, none_right ≥ 1.0, wrong ≤ 0.0; checker accepts_true ≥ 1.0, refuses_false ≥ 0.962), as owner paths in `scripts/merge_gate.py`; the readers may not run on live data unless their scores match the current prompts, role settings, and label sets and meet the floors |
| 2026-11-12 | ADR 0054 (agent's proposal; amends ADR 0010 and ADR 0043): the Gate 1 run of 2026-10-26 timed out, not failed: the stored-copy check reads copies one at a time, and the link check reached 2,031 of the stored ClinicalTrials.gov links (all working) in 45 minutes; `snapshots-verify --workers 8` with a 30-minute limit, and Gate 1's link step checks the links published records cite with ADR 0010's HTTP 200 rule (`check-links --published --strict`), narrowing ADR 0010's scope from every stored link; step limits add up to 145 of the job's 150 minutes |
Expand Down
24 changes: 21 additions & 3 deletions pipeline/catalyst/api/read.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from contextlib import contextmanager
from dataclasses import dataclass, replace
from datetime import date, timedelta
from typing import Any
from typing import Any, cast
from uuid import UUID

from pydantic import JsonValue
Expand All @@ -37,6 +37,7 @@
WindowView,
)
from catalyst.db.tables import TABLES
from catalyst.extract.agent import has_words
from catalyst.logs import log_event
from catalyst.schemas import (
Catalyst,
Expand Down Expand Up @@ -73,6 +74,7 @@
OPEN_STATUSES = (CatalystStatus.UPCOMING, CatalystStatus.SLIPPED)
BACKED_FIELDS = tuple(f.value for f in CatalystField if f is not CatalystField.DRUGS)
DRUGS = CatalystField.DRUGS.value
INDICATION = CatalystField.INDICATION_ID.value
TRIALS = "trials" # a source's `fields` entry for the quote that names a trial (ADR 0039)
DATE_FIELDS = frozenset(f.value for f in CatalystField if f.value.startswith("window_"))
MAX_LIMIT = 200
Expand Down Expand Up @@ -283,7 +285,7 @@ def _details(session: Session, rows: Sequence[Row[Any]]) -> list[CatalystDetail]
ids = [row.id for row in rows]
changes: dict[UUID, list[Row[Any]]] = defaultdict(list)
for change in session.execute(
select(_CHANGE, *_SOURCE_COLUMNS)
select(_CHANGE, *_SOURCE_COLUMNS, _CLAIM.c.value.label("claim_value"))
.join(_CLAIM, _CLAIM.c.id == _CHANGE.c.claim_id)
.join(_DOC, _DOC.c.id == _CLAIM.c.source_document_id)
.where(_CHANGE.c.catalyst_id.in_(ids))
Expand Down Expand Up @@ -379,7 +381,7 @@ def _build(
catalyst_type=row.catalyst_type,
title=row.title,
company=companies[row.company_id],
indication=indications.get(row.indication_id) if row.indication_id else None,
indication=_shown_indication(indications.get(row.indication_id), latest.get(INDICATION)),
drugs=[DrugView(id=link.drug_id, name=link.preferred_name, role=link.role) for link in links],
trials=[trial for _, trial in trials],
window=WindowView(
Expand All @@ -396,6 +398,22 @@ def _build(
)


def _shown_indication(indication: IndicationView | None, change: Row[Any] | None) -> IndicationView | None:
"""Return the indication named in its own source's words, not the MeSH name (ADR 0053, decision 2).

The words are those the claim behind the latest `indication_id` change kept from its sentence;
the MeSH name is shown only when that claim kept none.
"""
if indication is None or change is None:
return indication
stored: object = change.claim_value
value = cast(dict[str, object], stored) if isinstance(stored, dict) else {}
words = value.get("words")
if isinstance(words, str) and words.strip() and has_words(change.quote, words):
return indication.model_copy(update={"name": words})
return indication # words the cited quote does not hold are never shown


def drugs_value(links: Sequence[Row[Any]]) -> list[JsonValue]:
"""Return the links as a `drugs` change states them: `{drug_id, role}` pairs, sorted (ADR 0025)."""
pairs = sorted({(str(link.drug_id), str(link.role)) for link in links})
Expand Down
44 changes: 41 additions & 3 deletions pipeline/catalyst/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
render_html,
render_page,
)
from catalyst.publish.indication import publish_indications
from catalyst.publish.run import decide_claims
from catalyst.publish.sanity import check_safe_to_write
from catalyst.publish.write import publish_claims
Expand Down Expand Up @@ -178,7 +179,9 @@ def _parser() -> argparse.ArgumentParser:
"extract-catalysts",
help="Ask the extractor for the catalysts in stored press releases (costs money).",
)
extract.add_argument("--limit", type=int, default=50, help="Most documents to read in this run.")
extract.add_argument(
"--limit", type=_at_least_one, default=50, help="Most documents to read in this run."
)
extract.add_argument(
"--since", type=date.fromisoformat, default=None, help="Earliest publication date, YYYY-MM-DD."
)
Expand Down Expand Up @@ -599,6 +602,7 @@ def _extract(args: argparse.Namespace) -> int:
second = {"error": f"{type(error).__name__}: the second reading did not finish"}
decisions = _decide_after(sessionmaker(engine), store, checks)
published = _publish_after(sessionmaker(engine), store, decisions, config.roles[VERIFIER_ROLE].model)
indications = _indications_after(sessionmaker(engine), store, client, published, args.limit)
finally:
engine.dispose()
if args.out is not None:
Expand All @@ -610,10 +614,11 @@ def _extract(args: argparse.Namespace) -> int:
"second_reading": second,
"decisions": decisions,
"published": published,
"indications": indications,
}
print(json.dumps(summary, indent=2))
second_failed = "error" in second or second.get("documents_failed") or second.get("stopped_early")
gate_failed = "error" in decisions or "error" in published
gate_failed = "error" in decisions or "error" in published or "error" in indications
failed = summary["documents_failed"] or summary["stopped_early"] or second_failed or gate_failed
return 1 if failed else 0

Expand Down Expand Up @@ -677,6 +682,38 @@ def _publish_after(
return {"error": f"{type(error).__name__}: publishing did not finish"}


def _indications_after(
session_factory: sessionmaker[Session],
store: SnapshotStore,
client: ModelClient,
published: dict[str, Any],
limit: int,
) -> dict[str, Any]:
"""Find newly published catalysts' indications, then write the confirmed ones (ADR 0053).

After the writer, so new catalysts are asked about the same day; skipped when publishing did
not finish. A blocked gate or a role at its cap is reported, not an error: the catalysts
simply wait for a later run.
"""
if "error" in published or "skipped" in published:
return {"skipped": "publishing did not finish, so no indication was asked for"}
failures = (SQLAlchemyError, SnapshotStoreError, httpx.HTTPError, SyntaxError, ValueError, OSError)
result: dict[str, Any] = {}
try:
result["read"] = indicate_catalysts(
session_factory, store, client, limit=limit, now=_utc_now
).summary()
except failures as error:
result["read"] = {"error": f"{type(error).__name__}: the indication readers did not finish"}
try: # its own step, so finds stored by earlier runs are written even when the readers fail
result |= publish_indications(session_factory, store, _utc_now).summary()
except failures as error:
result["write"] = {"error": f"{type(error).__name__}: the indication pass did not finish"}
if "error" in result["read"] or "write" in result:
result["error"] = "the indication step did not finish"
return result


def _resolve(_args: argparse.Namespace) -> int:
try:
database = settings.database_url()
Expand Down Expand Up @@ -776,9 +813,10 @@ def _publish_claims(args: argparse.Namespace) -> int:
try:
model = load_config().roles[VERIFIER_ROLE].model
report = publish_claims(sessionmaker(engine), store, _utc_now, verifier_model=model)
indications = publish_indications(sessionmaker(engine), store, _utc_now)
finally:
engine.dispose()
print(json.dumps(report.summary(), indent=2))
print(json.dumps(report.summary() | indications.summary(), indent=2))
return 0


Expand Down
27 changes: 20 additions & 7 deletions pipeline/catalyst/extract/indicate.py
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,19 @@ def _tasks(session: Session, report: IndicationReport, models: tuple[str, str])
return tasks


def _load_index(session_factory: sessionmaker[Session], store: SnapshotStore) -> MeshIndex | str:
"""Return the stored MeSH index, or why the job cannot read it, so the run stops asking."""
try:
with session_factory() as session:
index = stored_index(session, store)
except (SnapshotStoreError, httpx.HTTPError, SyntaxError, ValueError) as error:
# A storage error, a changed copy, or a file that is not MeSH XML (review of PR 128).
return f"the stored MeSH file could not be read ({type(error).__name__})"
if index is None:
return "no MeSH descriptor file is stored yet; run `catalyst collect --sources mesh`"
return index


def indicate_catalysts(
session_factory: sessionmaker[Session],
store: SnapshotStore,
Expand All @@ -315,16 +328,16 @@ def indicate_catalysts(
report = IndicationReport(blocked=gate())
if report.blocked:
return report # no model is called without scores that meet their floors
if index is None:
with session_factory() as session:
index = stored_index(session, store)
if index is None:
report.blocked = ["no MeSH descriptor file is stored yet; run `catalyst collect --sources mesh`"]
return report
models = (client.model_for(ROLE), client.model_for(VERIFIER_ROLE))
provider = load_config().roles[ROLE].provider
with read_session(session_factory) as session:
tasks = _tasks(session, report, models)
if not tasks:
return report # nothing to ask about, so the MeSH file is not read
loaded = index if index is not None else _load_index(session_factory, store)
if isinstance(loaded, str):
report.blocked = [loaded]
return report
for task in tasks[:limit]:
report.considered += 1
try:
Expand All @@ -334,7 +347,7 @@ def indicate_catalysts(
if page.differs:
outcome = Outcome(IndicationMiss.COPY_DIFFERS) # such a copy's claims go to a person
else:
outcome = find_indication(client, text, page.visible, task.drug, task.quote, index)
outcome = find_indication(client, text, page.visible, task.drug, task.quote, loaded)
except SpendCapReachedError as error:
report.considered -= 1
report.stopped = str(error)
Expand Down
Loading
Loading