Example notebooks using EXFOR fails due to fprbidden access

[tkittel]

@marquezj We currently have 4 notebooks grabbing data from EXFOR via the web API, and they consistently seem to die in CI, e.g. https://github.com/mctools/ncrystal/actions/runs/25317982331/job/74220185533

Investigating a bit, it seems that a URL like the following

https://nds.iaea.org/exfor/x4get?DatasetID=40207005&op=csv

works when I paste it in my browser, but if I try to grab it via wget it fails:

$> wget 'https://nds.iaea.org/exfor/x4get?DatasetID=40207005&op=csv'
--2026-05-04 14:40:00--  https://nds.iaea.org/exfor/x4get?DatasetID=40207005&op=csv
Resolving nds.iaea.org (nds.iaea.org)... 172.66.138.10, 172.66.133.69, 2606:4700:10::ac42:8545, ...
Connecting to nds.iaea.org (nds.iaea.org)|172.66.138.10|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2026-05-04 14:40:00 ERROR 403: Forbidden.

So a guess is that the exfor server started to make "automated" downloads forbidden.

Can we get around this somehow? Are there any licensing issues of downloading a CSV file manually and caching it locally (or in a notebook cell)?

[marquezj]

The user-agent string is the existing work around (and still works in my computer), but I will add a static copy of the data to catch the error.

Unfortunately there seems to be no proper API to access the data.

[tkittel]

To be clear: Interactively running the notebook, or pasting the above URL into my browser, works. What does not work in my case is the wget command in the terminal or running the notebook code in a github actions workflow.

Which is why my suspicion is that the exfor server is looking at some metadata in the HTTP request and deciding "this is a robot" and therefore returning a "forbidden" error.

[marquezj]

You need to set an user-agent string. It is actually "official work-around", as listed in the bottom of the web API documentation. But, yes, it does not work with wget.

What seems to help, although not 100% of the time, is adding --header="Accept: text/html":

wget -O beryllium.txt --header="Accept: text/html"  --user-agent="Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"  "https://nds.iaea.org/exfor/x4get?DatasetID=40207005&op=csv"

[tkittel]

Ok, so it is clear why it doesnt work then.

But in any case, it might be better to hardwire the data in a notebook cell and put a comment in the description for how to access the data. Because I have in the past also had annoying spurious CI failures triggered by what looks like intermittent exfor server outages (or perhaps simply network issues).

[marquezj]

Regarding the license, I cannot find one for EXFOR itself but derivative projects from IAEA like exfortables_py is CC-BY-SA-4.0, and EXFOR Entry File is CC-BY-4.0.

[marquezj]

I added a dump of the data as a backup in PR 7. In the future we might look into this and develop a more general way to pull data from EXFOR.

[tkittel]

Thanks for fixing this in mctools/ncrystal-notebooks#7 ! We do still need to do a proper rerunning of these notebooks in cleaner envs before closing here - let us see who has time to do it first, it is not so urgent that it can't wait a few days.