[8 points] Backfill missed benchmarks and prove they are searchable - #534
JunkaiWang-TheoPhy wants to merge 7 commits into
Conversation
Issue ktwu01#533 requires an exact-source repair path and user-visible search proof for missed benchmarks. Reserve a clearly marked draft lane while the implementation is prepared; the placeholder carries no benchmark data and makes no completion claim.\n\nConstraint: maintainers requested a bounded evidence-preserving backfill with clean-rebuild search proof\nRejected: modify generated data before the repair path exists | it would create an unverifiable and non-idempotent fix\nConfidence: high\nScope-risk: narrow\nDirective: Replace this placeholder with tested implementation before marking the PR ready\nTested: git diff --check\nNot-tested: implementation and full CI are intentionally pending
Issue ktwu01#533 needs a bounded repair path because forward-looking connector changes cannot replay old RSS or GitHub lookback windows. Add exact arXiv and GitHub fetchers and a repair-source command that runs normal scoring, preserves source timestamps, records actual discovery/retrieval time, and writes through the existing snapshot merge path.\n\nConstraint: repaired evidence must stay in Radar snapshots and remain distinct from catalog/model-card layers\nRejected: patch generated site data or broaden current lookback queries | neither is idempotent or historically auditable\nConfidence: medium\nScope-risk: moderate\nDirective: Extend repair-source with reviewed Hugging Face/DOI adapters before closing ktwu01#533\nTested: Python compile, Ruff check, Ruff format check, git diff check; full pytest pending local Python compatibility\nNot-tested: live exact-source API calls and clean full rebuild
|
Implemented the first exact-source repair lane in the draft PR: arXiv IDs and GitHub owner/repo IDs now fetch outside the daily lookback, pass through normal scoring, preserve source timestamps, and write through the existing idempotent snapshot merge path. Added focused source fixtures. This remains draft while Hugging Face/DOI support and clean-rebuild search assertions are added. |
|
Focused verification now passes with Python 3.12: 100 source/CLI tests, including the exact arXiv and GitHub repair fixtures; Ruff check/format and compile checks also pass. The PR remains draft because the remaining acceptance work is Hugging Face/DOI adapters, reviewed FORGE decision, and clean-rebuild QueryService search assertions. |
Forward discovery fixes cannot replay an old arXiv bulletin or a repository outside the daily GitHub window. Add a bounded repair-source command for arXiv, GitHub, Hugging Face, and DOI metadata; preserve source and repair timestamps; merge through the existing snapshot identity path; and make stable source IDs searchable in CLI and dashboard all-dates views. Backfill the three confirmed cases, including the linked FORGE artifact, with an auditable Radar-layer fixture.\n\nConstraint: repairs must preserve publication/creation dates and remain distinct from catalog, model-card, and score layers\nRejected: patch generated site files or re-date old records as new releases | neither proves provenance or survives rebuilds\nConfidence: medium\nScope-risk: moderate\nReversibility: clean\nDirective: Keep repair-source bounded to reviewed stable IDs and add a reviewed inclusion decision before importing future cases\nTested: 1201-test full suite, Ruff check/format, normalize-external, classify, build-data-release, QueryService name/ID searches, four real source repairs\nNot-tested: live Crossref and Hugging Face production repair calls
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
JunkaiWang-TheoPhy
changed the title
|
PR #534 is now ready for review. The complete clean-checkout sequence passed: Ruff, normalize-external, classify, build-data-release, and 1201 tests. Four confirmed missed records are searchable in Radar by name and stable source ID, and the repair path is idempotent. |
|
@codex review |
The upstream main branch advanced with the latest catalog, snapshot, and Crossref work while issue ktwu01#533 was being implemented. Integrate those changes, retain the exact-source repair path and stable-ID search coverage, and keep the repaired records in the current Radar snapshot history.\n\nConstraint: PR ktwu01#534 must be tested against the current main branch before review\nRejected: force-push the pre-merge branch unchanged | GitHub reported a real merge conflict and stale validation base\nConfidence: high\nScope-risk: moderate\nDirective: Re-run the full clean-checkout sequence after future upstream report or snapshot changes\nTested: 1213 pytest tests, Ruff check/format, normalize-external, classify, build-data-release, QueryService status and seven name/ID search checks\nNot-tested: GitHub Actions result for this merged head
|
Rebased onto current upstream main in 1160248 and resolved the merge conflict. The final clean sequence now passes 1213 tests; Ruff, normalize-external, classify, build-data-release, QueryService status, and seven name/ID search checks are green. The PR remains ready for maintainer review. |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1160248f5b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Repair-source now reuses the scoring pass's selection counters, stops when scoring rejects the fetched record, and searches the full snapshot history before writing so reruns cannot duplicate an existing repair. Regression tests cover counters, rejection, and idempotence.\n\nConstraint: Snapshot selection metadata describes the pass that produced it.\nRejected: Copy the newest snapshot counters | that falsely reports a one-record repair as a full crawl.\nConfidence: high\nScope-risk: moderate\nDirective: Preserve exact repair identity across all historical snapshots.\nTested: Ruff and CLI tests (30 passed; targeted rejection passed).\nNot-tested: Full CI rerun pending.\n
|
Follow-up on the exact-repair review (head 3eef717):
Ruff and CLI/query-focused tests pass. Please review the current head. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3eef7177cd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Exact repairs now strip arXiv version suffixes, reject future-dated API records before scoring, and construct Hugging Face model links without a datasets segment. Regression tests cover all three source-specific invariants.\n\nConstraint: Repair identities must match ordinary ingestion and never publish future evidence.\nRejected: Store caller spelling verbatim | versioned arXiv IDs would duplicate canonical records.\nConfidence: high\nScope-risk: moderate\nDirective: Keep source-specific public URLs aligned with their upstream route semantics.\nTested: Ruff and CLI/source tests (113 passed).\nNot-tested: Full CI rerun pending.\n
|
Additional source-specific fixes are now in head 1d5971b: versioned arXiv IDs canonicalize to the base identifier, future-dated exact records fail before scoring, and Hugging Face model links use the correct route. CLI/source tests pass (113). |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1d5971b327
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| print(f"Already repaired {item.source}:{item.source_id}; no snapshot written") | ||
| return |
There was a problem hiding this comment.
If the first invocation writes the snapshot but rebuild_dashboard fails—or if the generated dashboard is later removed—a retry finds the persisted identity and returns here without rebuilding anything. This leaves site/data/radar.json stale or absent even though the command reports the repair as already complete, so rerunning the operation cannot recover its derived search surface; perform the rebuild before returning from this no-op path.
AGENTS.md reference: AGENTS.md:L110-L114
Useful? React with 👍 / 👎.
|
|
||
| def fetch_doi_exact(source_id: str) -> RadarItem: | ||
| """Fetch one DOI's bibliographic record from Crossref.""" | ||
| doi = source_id.removeprefix("https://doi.org/").removeprefix("http://doi.org/").strip() |
There was a problem hiding this comment.
When a repair is requested with uppercase DOI characters, this preserves the caller's spelling even though normal fetch_crossref ingestion case-folds DOI IDs. If the lowercase record is in an older snapshot, the literal idempotence check misses it and QueryService/dashboard deduplication keys the two observations by their distinct source_id strings, exposing duplicate Radar results for the same case-insensitive DOI; normalize the DOI consistently before fetching and persisting it.
AGENTS.md reference: AGENTS.md:L83-L87
Useful? React with 👍 / 👎.
| url=url, | ||
| published_at=published_at, | ||
| updated_at=created_at, | ||
| summary=str(message.get("abstract") or "").strip(), |
There was a problem hiding this comment.
For DOI records whose Crossref abstract contains the usual JATS/XML markup, this exact-repair path stores the markup verbatim, unlike the normal Crossref connector's clean_card_text path. The tags consequently appear as literal prose in the dashboard/search description and become searchable scoring tokens; pass the abstract through the existing cleaner before constructing the item.
AGENTS.md reference: AGENTS.md:L5-L10
Useful? React with 👍 / 👎.
|
这个方向有用，但当前快照语义需要先修正再合并。 现在四条记录是在 9 月 5 日修复的，却直接改写了 请改为写入修复当天的新快照或单独、可累计的 repair 记录，保留 9 月 3 日原始快照及其统计不变。修好后请补一段简短中文说明，重点解释快照时间和幂等语义。 |
Exact-source repairs now write a new snapshot dated at repair time instead of mutating the historical target day. Same-day repairs accumulate their source IDs, while prior snapshots and their original selection metadata remain untouched.\n\nConstraint: Historical snapshot dates and pass-level counters are immutable evidence.\nRejected: Merge repairs into the newest historical snapshot | that changes the meaning of generated_at and loses repair provenance.\nConfidence: high\nScope-risk: moderate\nDirective: Keep repair_source_ids cumulative for same-day exact repairs.\nTested: Ruff and repair idempotence/date test.\nNot-tested: Full CI rerun pending.\n
|
@codex review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
2 participants
中文说明
改了什么
重点审核
验证
Closes #533