Script fails with authorization error during DataverseNL recursion

[pedrohserrano]

Summary:

Our script for extracting all dataverses/datasets from the Maastricht root dataverse in DataverseNL is now breaking.
This script used to work until December 2024 but now stops with an authorization-related error.

Error Message

ApiAuthorizationError: ERROR: GET - Authorization invalid https://dataverse.nl/api/v1/dataverses/XXXXX. MSG: User ... is not permitted to perform requested action.

⸻

Steps to Reproduce

1. Run the script `python generate_reports_dataverseNL.py` to recursively extract all children (dataverses and datasets) from the Maastricht root dataverse.
2. The API connection is successful, but it breaks when gathering the `dataverse_tree`
3. The minimal extraction is `tree_data = api.get_children("maastricht", children_types=['dataverses', 'datasets'])`

Gut feelings

• The script appears to break at a deeper level in the dataverse tree (possibly at the department level).
• It’s possible that permissions have changed, or that some subdataverses are now restricted.
• There may have been changes to the DataverseNL security or SSO integration since last year.

[pedrohserrano]

Update: Diagnostic Script to Identify Permission Issues

The following script (GPT-4.1 assisted) walks the entire Maastricht dataverse tree and logs exactly where access is denied.
This makes it easy to identify which specific subdataverse(s) are causing the authorization error, so we can request the necessary permissions or adapt the script accordingly.

import requests

API_TOKEN = "API___"  # Replace with your actual token
BASE_URL = "https://dataverse.nl/api/v1"
START_ALIAS = "maastricht"

headers = {"X-Dataverse-key": API_TOKEN}
forbidden_dataverses = []

def walk_dataverse(dataverse_id_or_alias, parent_title="", level=0):
    url = f"{BASE_URL}/dataverses/{dataverse_id_or_alias}/contents"
    r = requests.get(url, headers=headers)
    indent = "  " * level
    if r.status_code != 200:
        try:
            msg = r.json()
        except Exception:
            msg = r.text
        print(f"{indent}❌ Forbidden: {dataverse_id_or_alias} (parent: '{parent_title}') [{r.status_code}]: {msg}")
        forbidden_dataverses.append({
            "dataverse_id_or_alias": dataverse_id_or_alias,
            "parent_title": parent_title,
            "status_code": r.status_code,
            "msg": msg
        })
        return
    children = r.json().get("data", [])
    for child in children:
        if child.get("type") == "dataverse":
            child_id = child.get("id") or child.get("alias")
            child_title = child.get("title", "")
            print(f"{indent}✅ OK: {child_title} ({child_id})")
            walk_dataverse(child_id, parent_title=child_title, level=level+1)

walk_dataverse(START_ALIAS)

print("\n==== Forbidden Dataverses Detected ====")
for bad in forbidden_dataverses:
    print(bad)

Example Output

Running this script, I identified at least one problematic subdataverse where access was denied:

❌ Forbidden: 467746 (parent: 'Shedata - Lama Sultan') [401]: {'status': 'ERROR', 'message': 'User @... is not permitted to perform requested action.'}

Next Steps

• We are requesting the appropriate permissions for the affected subdataverse(s) (in this case, ID 467746 under 'Shedata - Lama Sultan') to the dataverse curators of SHE department.
• This approach can also be used for ongoing monitoring as dataverse permissions evolve.