Add OpenAIRE as a daily-discovery source - #614
lazizbekravshanov wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved moderate connector findings and a README count inconsistency must be addressed before approval.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds OpenAIRE Graph v3 as an optional daily-discovery source, integrating metadata parsing, backfill, scoring, provenance, dashboard labels, documentation, and tests.
Changes:
- Adds bounded OpenAIRE fetching, normalization, deduplication, and validation.
- Registers OpenAIRE across configuration and processing pipelines.
- Adds connector and backfill regression tests and updates source documentation.
File summaries
| File | Summary and review status |
|---|---|
tests/test_sources.py |
Tests OpenAIRE parsing, validation, failure handling, deduplication, and request behavior. |
tests/test_pipeline.py |
Tests publication-date backfill placement. |
src/benchmark_radar/sources.py |
Adds the connector and registrations. Six unresolved moderate findings concern missing results handling (3 votes), absent counter defaults (1), date-only future validation (1), metadata stored only in raw (1), and malformed author (1) and organization (1) name coercion. |
src/benchmark_radar/rubric.py |
Adds primary evidence-source credit. |
src/benchmark_radar/pipeline.py |
Enables OpenAIRE backfill. |
src/benchmark_radar/corpus.py |
Adds provenance ranking. |
site/assets/app.js |
Adds OpenAIRE source display mappings. |
README.zh-CN.md |
Updates Chinese source documentation and counts. |
README.md |
Updates English source documentation. Nit (1 vote): line 149 still reports 37 sources and 13 direct connectors, conflicting with the updated counts. |
config.yml |
Configures the optional OpenAIRE source and documents API behavior. |
Review details
Suppressed comments (6)
README.md:26
- Please update the detailed count below as well:
README.md:149still says 37 sources (13 direct connectors), while this changed claim is 38 and the registry now derives 14 connectors plus 24 feeds. Leaving both values makes the README self-contradictory.
web. It pulls evidence from 38 public sources every day, and keeps updating.
src/benchmark_radar/sources.py:1268
- These
or 0defaults turn an absent counter into an observed zero. The sparse-record test exercises exactly this path, butprinciple.md:104-105andprinciple.md:146-156require unknown measurements to remain unknown; otherwise snapshots and adoption scores falsely report zero citations/downloads/views. Include a metric only when its upstream key is present, while retaining an explicit numeric zero.
metrics={
"citations": float(citations.get("citationCount") or 0),
"downloads": float(usage.get("downloads") or 0),
"views": float(usage.get("views") or 0),
},
src/benchmark_radar/sources.py:1203
publicationDateis date-only, but this upper-bound check relies only on_reject_future, which allows a five-minute clock-skew tolerance. Near midnight, a row dated tomorrow is therefore accepted (andwindow_endalready includes tomorrow), violating the connector's future-date rejection contract. Compare the publication date to the collection date separately; keep the tolerance only for genuinely timestamped fields.
if published.date() < window_start or _reject_future(config, source_id, published):
src/benchmark_radar/sources.py:1269
- Assigning the upstream row to
rawdoes not preserve its extra metadata in published data:RadarItem.to_dict()removesrawand snapshots explicitly reject raw payloads, leaving only a hash. As a result, the non-DOI PIDs anddateOfCollectionasserted initem.rawnever reach snapshots/catalog output, which contradicts the issue/PR requirement to preserve upstream identifiers and dates; project the required fields into the public item contract instead of relying onraw.
raw=row,
src/benchmark_radar/sources.py:1070
str(...)converts malformed non-string name fields (for examplefullName: ["Ada"]) into Python-repr text, so the connector publishes['Ada']instead of surfacing the malformed author payload. ValidatefullName,name, andsurnameas strings before joining to preserve deposited names and the documented malformed-payload failure behavior.
full_name = str(author.get("fullName") or "").strip()
if full_name:
return full_name
parts = (str(author.get("name") or "").strip(), str(author.get("surname") or "").strip())
return " ".join(part for part in parts if part)
src/benchmark_radar/sources.py:1087
- The same coercion issue exists for organizations: a malformed
legalNameoracronymobject/list is stringified into a fabricated label rather than rejected. Validate these fields as strings before calling.strip()so source health exposes an incompatible row instead of publishing Python-repr organization names.
for entry in _openaire_rows(value, "organizations"):
name = str(entry.get("legalName") or entry.get("acronym") or "").strip()
if name:
names.append(name)
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| rows = _payload_dict(payload, "OpenAIRE").get("results") | ||
| if rows is None: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e230dae58
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "OpenReview", | ||
| "Semantic Scholar", | ||
| "Crossref", | ||
| "OpenAIRE", |
There was a problem hiding this comment.
Every OpenAIRE result now receives the 40-point primary-source evidence credit solely because it came through this connector, so an otherwise identical record from an unlisted source is ranked lower despite carrying the same attached evidence. Keep OpenAIRE as provenance, but do not use source membership as a trust tier in the discovery score.
AGENTS.md reference: AGENTS.md:L237-L241
Useful? React with 👍 / 👎.
| "openreview", | ||
| "semantic_scholar", | ||
| "crossref", | ||
| "openaire", |
There was a problem hiding this comment.
When simulate-history runs for dates before today, adding this source causes simulate_backfill to call fetch_openaire without _collection_now; that fetcher therefore sets toPublicationDate to the real current date and asks only for the first descending page. Once newer products exceed page_size, the requested historical products never enter the pool and the run reports OpenAIRE healthy while producing incomplete historical snapshots. Pass the last simulated date as the collection bound (or omit this connector from simulated backfill).
AGENTS.md reference: AGENTS.md:L93-L97
Useful? React with 👍 / 👎.
| I kept running into new benchmarks while doing benchmark research, so I built a | ||
| crawler that continuously collects benchmark-related signals from across the | ||
| web. It pulls evidence from 37 public sources every day, and keeps updating. | ||
| web. It pulls evidence from 38 public sources every day, and keeps updating. |
There was a problem hiding this comment.
This new headline says 38 sources, but the Abstract later in README.md still says “37 sources: 13 direct connectors and 24 first-party” (and README.zh-CN.md has the same stale claim). With OpenAIRE enabled, the direct-connector count is 14, so readers encounter contradictory public coverage figures in the same document.
AGENTS.md reference: AGENTS.md:L12-L17
Useful? React with 👍 / 👎.
2e230da to
f2a2e6c
Compare
|
Thanks. One of the three findings was real and is fixed in the pushed commit. The README source total was genuinely inconsistent: the opening line said 38 while the Abstract still said "37 sources: 13 direct connectors and 24 first-party". Both mentions now read 38 and 14, and The other two I do not think hold, and I would rather explain than change the code quietly. On bounding the query to the simulated date: On the evidence credit: |
f2a2e6c to
2c52539
Compare
2c52539 to
5266222
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5266222fba
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "Semantic Scholar": 2, | ||
| "OpenAlex": 2, | ||
| "Crossref": 2, | ||
| "OpenAIRE": 2, |
There was a problem hiding this comment.
When an OpenAIRE observation and a rank-1 connector resolve to the same artifact, build_corpus uses this new source-only rank at lines 312–318 to prevent the OpenAIRE title and URL from becoming the public entity representation. That creates a preferred trust tier based solely on connector membership; retain the provenance records but choose the entity representation without source-based priority.
AGENTS.md reference: AGENTS.md:L237-L240
Useful? React with 👍 / 👎.
OpenAIRE aggregates publications, datasets, software and other research products from repositories across Europe and beyond, including institutional repositories this project reaches through no other source (issue ktwu01#545). This uses the Graph API v3, the version that superseded v1 and v2 in July 2026; the older `/search/*` API was announced for phase-out on 2026-05-31. Reads need no credential. An anonymous caller gets 60 requests per hour against 7,200 for a registered token, so the connector issues one request per search phrase and stays well inside the anonymous budget. A filter value holding whitespace, parentheses or a bare logical operator has to reach v3 double-quoted or the request is answered with HTTP 400 rather than a search. Every phrase this project ships contains a space, so the connector quotes them and a value that is already quoted or parenthesised is left intact. A page that matched nothing arrives as `results: null`, which is an ordinary empty result rather than the broken payload it would otherwise be reported as. The window is `publicationDate` on both ends of the query and again on the parsed record, and the record is dated by that same field. `dateOfCollection` says when OpenAIRE indexed a product rather than when it appeared, and is routinely months later; dating by it would hide a product on its publication day and surface it on a day the connector's own window guard rejects. Records duplicate the Crossref and arXiv copies of the same artifact by design. The DOI travels with every record that has one and is reduced to its bare name first, because repositories deposit it as a link and as a `doi:` URI as well, and a link passed through would build an address resolving nowhere that shares no identity with the other copies. A product whose only openable location is an instance URL or a code repository keeps that as its address; one that names no openable location at all is dropped rather than published pointing at nothing. Registered in configuration (optional, non-fatal), the evidence rubric and provenance rank beside Crossref, simulated backfill, and the dashboard source-name maps. Simulated backfill now asks each connector for the span it is rebuilding. A connector takes its upper bound from `_collection_now` and returns one page of its newest matches; `simulate_backfill` passed no such value, so every backfill source queried up to real now and handed back rows published this week. The per-date filter then discarded all of them and the rows belonging inside the requested windows were never fetched, so a rebuilt day came out empty rather than visibly wrong. Passing the newest simulated date fixes the bound for every source in `BACKFILL_SOURCES`, not only this one. The single-page cap still stands, so a long span is sampled rather than exhaustive; what changes is that the sample comes from inside the span. Both README source counts move from 37 to 38, in the opening line and in the Abstract, which also splits the figure into 14 connectors and 24 feeds; `count_ingest_sources` derives all three from the fetchers and the first-party feed list, and the README test now holds both mentions to it rather than only the first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaUf4LgR9HogJ1DhXwjESy
5266222 to
c4acfad
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c4acfad0a6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "openreview", | ||
| "semantic_scholar", | ||
| "crossref", | ||
| "openaire", |
There was a problem hiding this comment.
When a simulated span has more than page_size matches for an OpenAIRE phrase, adding this source here makes simulate-history fetch only its first, newest-sorted page once and then distribute that incomplete pool across all days. The new _collection_now bound fixes the prior current-date case, but it does not paginate this capped source; older days can therefore be written without their OpenAIRE observations while the copied source health still reports the fetch as healthy. Paginate for backfill or exclude OpenAIRE from BACKFILL_SOURCES.
AGENTS.md reference: AGENTS.md:L93-L97
Useful? React with 👍 / 👎.
Graph v3 returns the abstract as `descriptions`, a list, and carries no singular `description` member, so every published record reached the site with an empty summary. The suite passed because its own fixture invented the singular key. `score_item` matches the taxonomy over title plus summary, so the loss degraded categorisation without failing anything. Checked against the live endpoint rather than a fixture: 8 of 8 sampled products expose `descriptions` and none expose `description`, and a product routinely carries `descriptions: null` where it was deposited without an abstract, which is an absent abstract rather than a payload the connector cannot read. Also corrects two claims the connector documented that the live API contradicts: an anonymous caller is budgeted 7,199 requests per hour in `x-ratelimit-limit` rather than 60, and an unquoted multi-word value is accepted rather than answered with HTTP 400. Quoting is still right, for precision: it narrows `LLM benchmark` from 3,021 matches to 213. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Agent annotation: Pushed The bug
The suite did not catch it because The fixRead Live check after the fix, for Fixtures now match the real payload shape, plus two regression tests: a multi-entry Two documented claims correctedBoth are contradicted by the live API:
VerificationFull Still needs a maintainerCI has never run on this PR. Every run sits at Two items left for the author, not changed here:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c841a81e5b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }, | ||
| **_request_options(config), | ||
| ) | ||
| rows = _payload_dict(payload, "OpenAIRE").get("results") |
There was a problem hiding this comment.
When OpenAIRE returns an object without results—for example, an error-shaped HTTP 200 response—.get("results") yields None, so this code treats the malformed payload like the documented empty results: null response. run_pipeline consequently records OpenAIRE as healthy with zero items, silently losing its evidence from the durable daily snapshot; distinguish an absent key from an explicitly null value and raise ConnectorPayloadError for the former.
AGENTS.md reference: AGENTS.md:L93-L97
Useful? React with 👍 / 👎.
`.get("results")` returned None for both, so an error-shaped HTTP 200 body
carrying no `results` at all was read as the documented empty page: the
connector returned nothing and source health reported the day healthy.
That is the silent failure `_payload_rows` raises on for every other
connector, and the reason this one could not use it -- v3 really does send
`results: null` for a page that matched nothing -- argues for separating
the two cases rather than merging them. An absent key now raises; the
documented null is still an ordinary empty page, which its own test holds.
The premise is the documented response shape rather than a live check,
since the endpoint is not reachable from this sandbox.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaUf4LgR9HogJ1DhXwjESy
ktwu01
left a comment
ktwu01
left a comment
There was a problem hiding this comment.
Thanks for the live API follow-up; the descriptions and missing-results fixes are sound. I still found a few data-contract blockers:
- Missing counters are published as zero. Please omit absent citation, download, and view metrics while preserving a real upstream zero.
- Non-DOI PIDs and
dateOfCollectionexist only inraw, whichRadarItem.to_dict()removes, so the required identifiers and dates do not reach durable snapshots. Please project them into the public record contract and test serialization. - Historical backfill reads only one newest-first page per phrase, so longer spans silently lose older OpenAIRE records while reporting healthy. Please paginate backfill or exclude this capped connector from it.
- Date-only rows can pass as not-future during the five minutes before midnight; nested non-string author or organization names are also converted to Python representation text. Please add direct guards and regression cases.
Both clean CI runs pass (1,418 tests on the head; 1,440 with current main) and the merge itself is conflict-free. Thank you.
Closes #545
What this gives the radar
daily-radar.ymlis untouched, so there is nothing to coordinate on credentials. A registered token raises the budget from 60 to 7,200 requests per hour if it is ever wanted; the connector does not need it.mainat 5d9e378, 10 files, 925 additions. Clean-worktree CI sequence passes; details under Verification.Two things to decide
count_ingest_sourcesinscripts/export_report_figure_data.pyderives, andtest_readmes_state_the_live_ingest_source_countnow holds both READMEs to it. The repository description on GitHub also says "37 public sources"; that is a setting rather than a file, so it is left for you.maincommit and both write 38. Whichever lands second conflicts on the two README lines and on adjacent registration lines insources.py,config.yml,pipeline.py,rubric.py,corpus.py,app.jsand the two test files. Every one is an adjacent-insertion conflict, resolved by keeping both sides and writing 39. I will rebase the second one the same day the first merges.Acceptance criteria from #545
GET https://api.openaire.eu/graph/v3/research-products, the Graph API v3 that superseded v1 and v2 in July 2026 (the older/search/*API was announced for phase-out on 2026-05-31); all five documented in theopenaireblock ofconfig.ymland in the connector docstringfetch_openairecarries the DOI and other PIDs, main title,publicationDateanddateOfCollection, authors as deposited, organisations by legal name, short name or acronym, description, instance URLs and code repository, plus the citation, download and view counters; no summary is synthesizedfromPublicationDate/toPublicationDatein the query and again on the parsed record; future-dated rejected; unreadable shapes raiseConnectorPayloadError, HTTP failuresRequestError, both surfaced as an optional, non-fatal sourceconfig.yml,SOURCE_FETCHERS/_PARSER_VERSION_METHODS/SOURCE_DEFAULT_METHODS,EVIDENCE_PRIMARY_SOURCES,PRIMARY_SOURCE_RANK,BACKFILL_SOURCES, both maps insite/assets/app.jstests/test_sources.pyandtests/test_pipeline.py, listed under Tests; CI under VerificationRecords merge only on the exact DOI, the key
dedupe_keysalready uses, so nothing here does fuzzy matching. Nothing is scraped behind authentication.How the connector behaves
mainTitlevalue holding whitespace, parentheses or a bare logical operator has to reach v3 double-quoted or the request is answered with HTTP 400 rather than a search. Every phrase this project ships contains a space, so the connector quotes them, escapes an embedded", and leaves a value that is already quoted or parenthesised intact. Without this the source would return nothing at all in production while looking configured.publicationDatebounds the query on both ends and dates the record.dateOfCollectionsays when OpenAIRE indexed a product rather than when it appeared, and is routinely months later; dating by it would hide a product on its publication day and surface it on a day the connector's own window guard rejects.doi:URI. A link passed through would buildhttps://doi.org/https://doi.org/..., which resolves nowhere and shares no identity with the Crossref or Zenodo copy of the same artifact, so the record would be counted twice.pageSizeis capped at 100 by the API and the connector clamps a larger configured value;sortBy=publicationDate DESCso a page holds the newest matches. One page per phrase, so a run costsmax_requestscalls. A page that matched nothing arrives asresults: null, which is treated as an ordinary empty result rather than a broken payload.resultsarray, or a malformedauthors,pids,instancesororganizationsshape is a parsing gap the dashboard should show, not a quietly thinner record.Wiring
config.yml: optionalopenaireblock (6 phrases,page_size: 50,max_requests: 6), with the semantics above documented in place.src/benchmark_radar/sources.py:fetch_openaireand its registrations.src/benchmark_radar/pipeline.py:BACKFILL_SOURCES; the publication date range makes a past window honestly re-derivable.src/benchmark_radar/rubric.py,src/benchmark_radar/corpus.py: primary evidence credit and provenance rank 2, beside Crossref.site/assets/app.js: both source-name maps.README.md,README.zh-CN.md: acknowledgement list and source count.Tests
tests/test_sources.py: a valid record (every preserved field and the exact bounded, quoted query), undated, out-of-window, future-dated, no openable location, instance and repository fallbacks, a non-openable repository, author and organisation shapes, malformed row shapes, cross-search DOI dedupe with newest-first ordering, limit truncation, thepageSizecap, a minimal product, the request budget, quoting of plain, parenthesised, already-quoted and embedded-quote phrases, and the DOI deposited bare, as a link and as adoi:URI. Plus OpenAIRE entries in the shared empty / malformed / HTTP-failure / no-synthesized-summary parametrizations and the collection-method default.tests/test_pipeline.py:test_simulate_backfill_places_an_openaire_product_on_its_publication_dayruns the real connector throughsimulate_backfill, so the connector's window key and the backfill placement key are checked against each other rather than separately.Each behaviour above that a test protects was checked by reverting it and confirming the test goes red, so the tests carry their regression rather than restating current output.
Verification
Clean-worktree run of the CI sequence (
git worktree add --detach, submodule initialized) on this exact head:ruff check .,ruff format --check .,normalize-catalog,classifyandbuild-data-releasepass;pytest -qreports 1384 passed, 1 failed.The one failure is
tests/test_briefing.py::test_zh_output_budget_covers_worst_case, which downloads tiktoken'so200k_basetable fromopenaipublic.blob.core.windows.net, a host the build sandbox's egress policy blocks. It fails identically on an untouchedmaincheckout in the same sandbox and passes on GitHub's runner.One honest caveat. The live OpenAIRE API was not reachable from the sandbox this was built in, so the request and response shapes were verified against OpenAIRE's own API contract tests plus two independent client libraries that agreed on the response shape, rather than a live call. The first daily workflow run after merge is the first live exercise; source health will show any surprise, and I will watch it.
Not in this PR
Related identifiers and
relationsstay inraw; carrying them as artifact URLs needs its own review, since any cited DOI would merge a record with everything it cites. The connector does not paginate past the first page per phrase, which the config block states.🤖 Generated with Claude Code
https://claude.ai/code/session_01UaUf4LgR9HogJ1DhXwjESy