Add DataCite as a daily-discovery source - #613
lazizbekravshanov wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ccf730b141
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Title-scoped, like the Crossref connector: the description | ||
| # field indexes every abstract that mentions a benchmark in | ||
| # passing, and the shared taxonomy filters the rest downstream. | ||
| "query": f"titles.title:({_datacite_query_terms(search)}) AND {window}", |
There was a problem hiding this comment.
DataCite’s query_string parser treats whitespace-separated terms as alternatives unless they are explicitly joined, so this makes configured searches such as LLM benchmark match titles containing only LLM or only benchmark. The latter admits unrelated benchmark deposits (for example, geodesy or toxicology benchmarks) into the daily feed, and the broad benchmark taxonomy then publishes them rather than filtering them out. Build the title expression with explicit AND operators (or a quoted phrase, if exact phrasing is intended) so it matches the configuration comment’s stated all-terms requirement.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🟡 Changes recommended
Seven unresolved moderate findings remain in the DataCite connector.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds DataCite as an optional daily-discovery source for DOI-bearing datasets, software, and reports.
Changes:
- Implements DataCite fetching, parsing, filtering, and deduplication.
- Integrates the source into backfill, scoring, provenance, and dashboard mappings.
- Updates configuration and documentation, with connector and backfill tests.
File summaries
| File | Reviewed changes |
|---|---|
tests/test_sources.py |
DataCite parsing, filtering, deduplication, and failure tests. |
tests/test_pipeline.py |
Registration-day backfill regression coverage. |
src/benchmark_radar/sources.py |
DataCite connector and registration; unresolved findings cover malformed fields, timestamp validation, and missing counters. |
src/benchmark_radar/rubric.py |
DataCite evidence-scoring integration. |
src/benchmark_radar/pipeline.py |
DataCite backfill integration. |
src/benchmark_radar/corpus.py |
DataCite provenance ranking. |
site/assets/app.js |
Dashboard source labels and method mappings. |
README.zh-CN.md |
Chinese source documentation and count updates. |
README.md |
English source documentation and count updates. |
config.yml |
DataCite search configuration and API semantics. |
Review details
Suppressed comments (6)
src/benchmark_radar/sources.py:1163
- Using
or []means an explicitly malformed but falseytitlesvalue (for example{},"", or0) bypasses_datacite_rowsand is treated as a missing title, so the connector silently drops the row instead of raisingConnectorPayloadErrorfor an unreadable shape. Use a default that applies only when the field is absent.
title = _datacite_title(attributes.get("titles") or [])
src/benchmark_radar/sources.py:1177
- The same
or []coercion lets falsey malformedcreatorsvalues bypass_datacite_rows, so a bad object or scalar is accepted as an empty author list and the source remains healthy. This contradicts the connector's documented shape validation; default only when the key is absent.
for creator in _datacite_rows(attributes.get("creators") or [], "creators"):
src/benchmark_radar/sources.py:1221
- The
or []coercion also hides falsey malformeddescriptionsvalues, treating them as missing instead of surfacing aConnectorPayloadError. That makes the promised malformed-payload health signal incomplete; use a missing-key default rather than truthiness.
_datacite_description(attributes.get("descriptions") or [])
src/benchmark_radar/sources.py:1071
- This coercion lets malformed DataCite title values through: a non-string such as an object or array is converted to its Python representation and published as the title.
_datacite_rowsonly validates the containing row, so this violates the connector's fail-loud handling for unreadable shapes; validate thetitlefield's type before normalizing it.
(str(row.get("titleType") or "").strip(), str(row.get("title") or "").strip())
src/benchmark_radar/sources.py:1085
- The description value is also stringified without a type check, so a malformed object/list becomes user-visible summary text instead of causing the DataCite source to fail visibly. Require a string description (or reject the row) before calling
strip()so malformed upstream shapes cannot be silently published.
str(row.get("description") or "").strip(),
src/benchmark_radar/sources.py:1164
_optional_dateaccepts a date-only value by assigning midnight UTC, butregisteredis the timestamp used for the second-precise collection window. A malformed/date-only response can therefore be admitted at an invented instant and pass the lower-bound check; reject non-datetimeregisteredvalues instead of treating them as exact timestamps.
registered = _optional_date(attributes.get("registered"))
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| metrics={ | ||
| "citations": float(attributes.get("citationCount") or 0), | ||
| "downloads": float(attributes.get("downloadCount") or 0), | ||
| "views": float(attributes.get("viewCount") or 0), | ||
| }, |
ccf730b to
05a98ef
Compare
|
Thanks to both reviewers. Six of the seven findings were real and are fixed in the pushed commit. The Codex finding on the title query was correct and was the most serious: the config block documented an all-terms rule that the code did not implement, because Copilot's five findings on shape handling were also correct. One finding I do not think holds, and I would rather say so than change the code quietly. On |
05a98ef to
0196356
Compare
0196356 to
e953269
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e9532697a7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "openreview", | ||
| "semantic_scholar", | ||
| "crossref", | ||
| "datacite", |
There was a problem hiding this comment.
Adding DataCite here makes simulate_backfill call fetch_datacite once with only the oldest lower bound and no _collection_now. The connector therefore builds its registered query through the real wall-clock time, then asks for only 50 rows per phrase ordered by -created; for a historical backfill, those current rows are discarded by the later per-day visibility filter while the rows registered in the requested historical windows are not returned. This silently produces incomplete DataCite observations in simulated snapshot history; bound the fetch to max(dates) (or otherwise retrieve the requested windows) before including it as backfill-capable.
AGENTS.md reference: AGENTS.md:L93-L97
Useful? React with 👍 / 👎.
e953269 to
ac9f3a2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac9f3a255c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| updated = _optional_date(attributes.get("updated")) | ||
| if registered < window_start or _reject_future(config, doi, registered, updated): | ||
| continue |
There was a problem hiding this comment.
When simulate_backfill fetches a past span, it sets _collection_now to the last simulated date, but this call also passes DataCite's mutable updated timestamp to _reject_future. Therefore a DOI registered within (for example) a July simulation window but edited in August is dropped before it can be assigned to its registration day, despite the connector later normalizing updated_at to registered. This makes simulate-history silently omit valid DataCite evidence whenever the current metadata has been edited after the requested period; only checking the immutable registration timestamp for this connector would preserve the historical snapshots.
AGENTS.md reference: AGENTS.md:L93-L97
Useful? React with 👍 / 👎.
DataCite is the other DOI registration agency. Datasets, software and reports deposited with Zenodo, Figshare, Dryad and institutional repositories now have an independent path into the daily feed, beside the Crossref connector that covers publisher-deposited DOIs (issue ktwu01#544). The window is the `registered` timestamp, the moment the DOI started to resolve, applied on both ends of the query and again on the parsed record. `published` is often a bare year and `created` can predate registration by however long a deposit sat in draft, so neither bounds a daily window. A record is dated by that same instant rather than by DataCite's mutable `updated`: recency scoring and simulated backfill both read `updated_at`, so keying it on a metadata edit would hand a long-stale DOI a new record's recency score and place it in a backfilled day this connector's own window guard rejects. Search phrases reach a query_string parser whose default operator is OR, so a bare "LLM benchmark" matched a title carrying only "benchmark" and admitted geodesy and toxicology deposits that this project's broad benchmark taxonomy then published. The terms are joined with an explicit AND, which is the all-terms rule the config block documents. Reserved characters are escaped first, because an unescaped colon in a phrase such as "Benchmark: agents" reads as a clause on an unmapped field and returns zero rows while source health still reports the source healthy. Shapes the connector cannot read are reported rather than absorbed. An array field defaults only when it is absent or null; a present but unreadable `{}`, `""` or `0` used to pass as an empty list, so a row was silently dropped or published without its authors while the source still looked healthy. A title or description that is not a string raises instead of being stringified, which had published a Python repr as upstream text that no reader could trace back to the deposit. Only an explicitly personal name is reordered out of `Family, Given`: `nameType` is optional in the DataCite schema, and assuming a missing one means "personal" would publish `University of California, Berkeley` as `Berkeley University of California`. Registered in configuration (optional, non-fatal), the evidence rubric and provenance rank beside Crossref, simulated backfill, and the dashboard source-name maps. Simulated backfill now asks each connector for the span it is rebuilding. A connector takes its upper bound from `_collection_now` and returns one page of its newest matches; `simulate_backfill` passed no such value, so every backfill source queried up to real now and handed back rows published this week. The per-date filter then discarded all of them and the rows belonging inside the requested windows were never fetched, so a rebuilt day came out empty rather than visibly wrong. Passing the newest simulated date fixes the bound for every source in `BACKFILL_SOURCES`, not only this one. The single-page cap still stands, so a long span is sampled rather than exhaustive; what changes is that the sample comes from inside the span. Only `registered` is checked against that collection instant. DataCite stamps `updated` on every metadata edit and the connector deliberately refuses to date a record by it, so gating the record on it meant that once the fetch was bound to the end of a simulated span, a DOI registered inside the window and edited afterwards was dropped before it could reach its registration day. An edit after the span is ordinary history, not a future timestamp; the sibling Crossref connector checks its own `published` alone for the same reason, and the deposited `updated` still travels in `raw`. Both README source counts move from 37 to 38, in the opening line and in the Abstract, which also splits the figure into 14 connectors and 24 feeds; `count_ingest_sources` derives all three from the fetchers and the first-party feed list, and the README test now holds both mentions to it rather than only the first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaUf4LgR9HogJ1DhXwjESy
ac9f3a2 to
74cf8e6
Compare
Closes #544
What this gives the radar
daily-radar.ymlis untouched, so there is nothing to coordinate on credentials.mainat 5d9e378, 10 files, 699 additions. Clean-worktree CI sequence passes; details under Verification.Two things to decide
count_ingest_sourcesinscripts/export_report_figure_data.pyderives, andtest_readmes_state_the_live_ingest_source_countnow holds both READMEs to it. The repository description on GitHub also says "37 public sources"; that is a setting rather than a file, so it is left for you.maincommit and both write 38. Whichever lands second conflicts on the two README lines and on adjacent registration lines insources.py,config.yml,pipeline.py,rubric.py,corpus.py,app.jsand the two test files. Every one is an adjacent-insertion conflict, resolved by keeping both sides and writing 39. I will rebase the second one the same day the first merges.Acceptance criteria from #544
GET https://api.datacite.org/dois; documented in thedataciteblock ofconfig.ymland in the connector docstringfetch_datacitecarries DOI, titles,registered/published/updated, creators with affiliations, abstract, landing URL, and citation, download and view counts as deposited; no summary is synthesizedregistered:[since TO now+tolerance]in the query and again on the parsed record; future-dated on either timestamp rejected; unreadable shapes raiseConnectorPayloadError, HTTP failuresRequestError, both surfaced as an optional, non-fatal sourceconfig.yml,SOURCE_FETCHERS/_PARSER_VERSION_METHODS/SOURCE_DEFAULT_METHODS,EVIDENCE_PRIMARY_SOURCES,PRIMARY_SOURCE_RANK,BACKFILL_SOURCES, both maps insite/assets/app.jstests/test_sources.pyandtests/test_pipeline.py, listed under Tests; CI under VerificationRecords merge only on the exact DOI, which is the key
dedupe_keysalready uses, so nothing here does fuzzy matching. Nothing is scraped behind authentication.How the connector behaves
registered, the moment the DOI started to resolve.publishedis often a bare year andcreatedcan predate registration by a long draft period, so neither bounds a daily window. A record is dated by that same instant:updated_atis the registration timestamp, not DataCite's mutableupdated, because recency scoring and simulated backfill both readupdated_at, and keying it on an edit would hand a years-old DOI a new record's recency score and place it in a backfilled day the connector's own window guard rejects. The depositedupdatedstays inraw. This matches the Crossref sibling, which dates records bypublished.findableDOIs are used. An anonymous read only ever returns that state; the connector skips any other state as a guard for a credentialed run. A deposit registered first and made findable later is therefore collected on its registration date or not at all, and the config comment says so.query_stringclause. An unescaped colon in a phrase such asBenchmark: agentsreads as a lookup on a field the non-dynamic mapping lacks and returns zero rows while source health still reports the source healthy. The shipped phrases contain no reserved characters, so the emitted query is unchanged today.Family, Given.nameTypeis optional in the DataCite schema, and assuming a missing one means "personal" would publishUniversity of California, BerkeleyasBerkeley University of California.Abstractis preferred over a technical note.page[size]is capped at 1,000 by the API andpage[cursor]exists for deeper pulls; the connector takes one page per phrase, so a run costsmax_requestscalls.registeredis not a server-side sort, so results are ordered by-createdupstream and re-sorted by registration here.dataarray, or a malformedattributes,titles,descriptions,creatorsoraffiliationshape is a parsing gap the dashboard should show, not a quietly thinner record.Wiring
config.yml: optionaldataciteblock (6 phrases,page_size: 50,max_requests: 6), with the semantics above documented in place.src/benchmark_radar/sources.py:fetch_dataciteand its registrations.src/benchmark_radar/pipeline.py:BACKFILL_SOURCES; the registered date range makes a past window honestly re-derivable.src/benchmark_radar/rubric.py,src/benchmark_radar/corpus.py: primary evidence credit and provenance rank 2, beside Crossref.site/assets/app.js: both source-name maps.README.md,README.zh-CN.md: acknowledgement list and source count.Zenodo mints its DOIs through DataCite, so the same deposit can arrive from both connectors; both records carry the DOI and merge on it.
Tests
tests/test_sources.py: a valid record (every preserved field and the exact bounded query), undated, out-of-window, future-dated on either timestamp, non-findable state, untitled rows, a title with only a typed entry, name and affiliation shapes, six malformed row shapes, cross-search DOI dedupe with newest-first ordering, limit truncation, a minimal deposit, and query escaping. Plus DataCite entries in the shared empty / malformed / HTTP-failure / no-synthesized-summary parametrizations and the collection-method default.tests/test_pipeline.py:test_simulate_backfill_places_a_datacite_doi_on_its_registration_dayruns the real connector throughsimulate_backfill, so the connector's window key and the backfill placement key are checked against each other rather than separately.Each behaviour above that a test protects was checked by reverting it and confirming the test goes red, so the tests carry their regression rather than restating current output.
Verification
Clean-worktree run of the CI sequence (
git worktree add --detach, submodule initialized) on this exact head:ruff check .,ruff format --check .,normalize-catalog,classifyandbuild-data-releasepass;pytest -qreports 1358 passed, 1 failed.The one failure is
tests/test_briefing.py::test_zh_output_budget_covers_worst_case, which downloads tiktoken'so200k_basetable fromopenaipublic.blob.core.windows.net, a host the build sandbox's egress policy blocks. It fails identically on an untouchedmaincheckout in the same sandbox and passes on GitHub's runner.One honest caveat. The live DataCite API was not reachable from the sandbox this was built in, so the request and response shapes were verified against DataCite's own API implementation (lupo:
openapi.yaml, the/doiscontroller, the Elasticsearch mapping and the JSON:API serializer) rather than a live call. The first daily workflow run after merge is the first live exercise; source health will show any surprise, and I will watch it.Not in this PR
Related identifiers stay in
raw. CarryingIsSupplementToorIsVersionOflinks as artifact URLs would merge a dataset with its paper, but any cited DOI would merge it with everything it cites, so that needs its own review. The same applies to repositories that register a concept DOI alongside each version DOI: both rows are findable and only a long shared title currently merges them. The connector does not paginate past the first page per phrase, which the config block states.🤖 Generated with Claude Code
https://claude.ai/code/session_01UaUf4LgR9HogJ1DhXwjESy