Add scheduled adapter ingestion workflow by DeepLumiere · Pull Request #184 · evaleval/every_eval_ever

DeepLumiere · 2026-06-11T15:34:43Z

No description provided.

This script runs all Every Eval Ever adapters, manages their execution, and uploads results to Hugging Face.

Updated the repository ID and modified the upload process to use upload_large_folder. Added checks for input requirements and improved error handling during adapter processing.

Sanitize filenames to avoid issues with Hugging Face Hub by replacing invalid characters and handling names starting with 'x00'.

Updating files

Added 'env' parameter to get_dynamic_args function to allow passing environment variables for subprocess execution. Updated subprocess calls to use the provided environment.

Refactor adapter staleness check and input requirement parsing. Removed unused header checks and simplified logic for determining if an adapter is stale.

Refactor run_adapters.py by removing unused imports, simplifying functions, and updating the download_hf_json function to remove revision parameter. Additionally, streamline adapter processing and improve header checks.

tommasocerruti

@DeepLumiere Thank you for the great work! 🚀 Could you please:

Retitle this PR to something like “Add scheduled adapter ingestion workflow”;
Mark it as a draft while adapter coverage and failure handling are still wip;
Add a living description with the goal, design, and an adapter status table that you can update as support improves;
Split the unrelated changes (e.g., ARC adapter, helper change, ...) into separate PRs. Unless there is a strong reason, let's keep it focused on the scheduled adapter runner;
Address the inline comments I left.

tommasocerruti · 2026-07-03T14:01:45Z

+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version-file: "pyproject.toml"


Would it be safer to pin Python 3.12 here? Since the constraint in pyproject.toml is >=3.12, the latest run selected Python 3.14.6, which appears to have caused the datasets/dill failures in the SWE adapters.

Got it, thank you!

tommasocerruti · 2026-07-03T14:03:23Z

+
+    return False
+
+def get_input_requirement(adapter_path: Path):


I’m a little concerned about discovering adapter arguments by parsing their source code with regexes. This can easily miss multiline constants or slightly different CLI conventions. Would an explicit adapter config be simpler and more reliable? An alternative idea could be a common CLI contract for every adapter.

I’m a little concerned about discovering adapter arguments by parsing their source code with regexes. This can easily miss multiline constants or slightly different CLI conventions. Would an explicit adapter config be simpler and more reliable? An alternative idea could be a common CLI contract for every adapter.

A contract I feel that was discussed as a base class of adapter so that future work doesn't need major adaptation with Leshem Choshen and Harsha Nelaturu. I'll check it out whether something like that already exists, else try to make it.

tommasocerruti · 2026-07-03T14:04:18Z

+    DATA_DIR.mkdir(exist_ok=True)
+
+    # Download existing state
+    stats = download_hf_json("adapter_stats.json", {})


Is this path intentional? We write the file as data/adapter_stats.json below, but read adapter_stats.json from the repository root here. Also, this reads from the default branch while the upload creates a PR, so I’m not sure the state from one run will be available to the next one.

As of now decision making as per previous one is to be decided, weekly or daily what to do with heavy ones and all, any suggestions?

tommasocerruti · 2026-07-03T14:06:14Z

+                "time_s": elapsed,
+                "size_mb": dir_size_mb,
+                "last_success_ts": time.time(),
+                "last_failed": False


Could we only mark an adapter successful once validation has produced at least one valid file and no invalid files? At the moment, missing validator output or an incorrect output directory results in 0 valid, 0 failed, but the adapter is still recorded as successful.

Got it, I'll update that

tommasocerruti · 2026-07-03T14:07:06Z

+                folder_path=".",
+                repo_type=REPO_TYPE,
+                allow_patterns=["data/**"],
+                create_pr=True 


How are subsequent runs expected to update the same Hugging Face PR? Would it make sense to keep the returned PR revision after the first upload and target that revision on later runs? It would also be useful to log the returned commit/PR URL so we can verify what was updated.

tommasocerruti · 2026-07-03T14:08:32Z

+            )
+            print("Upload complete!")
+        except Exception as e:
+            print(f"Upload failed: {e}")


Should this failure be re-raised or should the script exit nonzero at the end? Right now an upload failure (as well as the adapter failures collected above) can still leave the GitHub Actions run green, which makes the scheduled job look healthier than it is.

# Conflicts: # utils/scripts/run_adapters.py

Refactor run_adapters.py for improved readability and functionality. Changes include updating function signatures, enhancing error handling, and optimizing file operations.

feat: add Mercor evaluation adapter (evaleval#199)

DeepLumiere added 4 commits June 10, 2026 19:37

Update source URL and modify log file structure

f872ce5

Fix UUID handling and improve log writing format

b92fab5

added aggregate and instance level jsonl

83007f4

Refactor log writing to use UUID filename segments

fa48968

DeepLumiere mentioned this pull request Jun 13, 2026

[adapter] Update arc_agi adapter #162

Open

borgr assigned nelaturuharsha Jun 15, 2026

DeepLumiere added 24 commits June 18, 2026 18:57

Merge branch 'evaleval:main' into main

419d029

Add script to run and manage Every Eval Ever adapters

0b09425

This script runs all Every Eval Ever adapters, manages their execution, and uploads results to Hugging Face.

Refactor run_adapters.py for repo ID and upload changes

42618d6

Updated the repository ID and modified the upload process to use upload_large_folder. Added checks for input requirements and improved error handling during adapter processing.

First Run of Adapter Cron

162f85d

Add environment to adapter cronjob workflow

2e2b85c

Enhance filename sanitization in io.py

ed82ecb

Sanitize filenames to avoid issues with Hugging Face Hub by replacing invalid characters and handling names starting with 'x00'.

Fix upload function to use upload_folder

9cbffab

Change cron schedule to run weekly on Sundays

6dbe1af

Update repository ID and add revision parameter

2595b89

Merge pull request #1 from evaleval/main

0482799

Updating files

Update run_adapters.py

7d38126

Update run_adapters.py

5c025ce

Update run_adapters.py

1a7ba4c

Update argument parser description in run_adapters.py

c8727c6

Add files via upload

d390cc2

Update run_adapters.py

fc0f861

Update run_adapters.py

3005137

Modify get_dynamic_args to accept environment variables

7d65667

Added 'env' parameter to get_dynamic_args function to allow passing environment variables for subprocess execution. Updated subprocess calls to use the provided environment.

Update run_adapters.py

9744e7e

Refactor adapter staleness logic and input parsing

10229ab

Refactor adapter staleness check and input requirement parsing. Removed unused header checks and simplified logic for determining if an adapter is stale.

Update run_adapters.py

ecc370b

Update run_adapters.py

a95103e

Update run_adapters.py

abcb627

Refactor run_adapters.py for clarity and efficiency

f4fec94

Refactor run_adapters.py by removing unused imports, simplifying functions, and updating the download_hf_json function to remove revision parameter. Additionally, streamline adapter processing and improve header checks.

tommasocerruti reviewed Jul 3, 2026

View reviewed changes

DeepLumiere changed the title ~~arc_agi adapter for flat storage with manifest and instance_level indexes for hf~~ Add scheduled adapter ingestion workflow Jul 3, 2026

DeepLumiere requested a review from tommasocerruti July 4, 2026 05:10

Dudecomputerscience and others added 23 commits July 4, 2026 20:45

feat: add cron job and script for running adapters

3ba7388

Merge remote-tracking branch 'origin/main'

21f59de

# Conflicts: # utils/scripts/run_adapters.py

feat: add configuration files and scripts for data processing

d49c8df

Update adapter_cron.yml

a8b7ed1

Delete .github/workflows/adapter_cronjob.yml

b47f6fa

Refactor run_adapters.py for clarity and efficiency

b868b7e

Refactor run_adapters.py for improved readability and functionality. Changes include updating function signatures, enhancing error handling, and optimizing file operations.

Update adapter.py

b4fa9f8

Change import path for eval_types module

a77d15f

Increase thresholds for heavy adapters classification

2ffe902

Increase HEAVY_TIME_S threshold from 240 to 900

0c409bb

add output directory argument to adapter scripts and update dependencies

84f8056

feat: add input option to force run all adapters in cron job

01d1834

Update cron job description for force_all input

657a3a7

Fixed dataset link

05aac8a

Upgrade to run_adapters.py

38bf464

Merge remote-tracking branch 'origin/main'

0eeeb7d

Fork Update

9ea6216

feat: add Mercor evaluation adapter (evaleval#199)

Upgrade to run_adapters.py

9c84a92

Merge remote-tracking branch 'origin/main'

f70b24f

Increased limits

5089eae

Upgrade to run_adapters.py

a06ab07

Merge remote-tracking branch 'origin/main'

c57bfe1

Added Mercor See Crate

cf36ec8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add scheduled adapter ingestion workflow#184

Add scheduled adapter ingestion workflow#184
DeepLumiere wants to merge 51 commits into
evaleval:mainfrom
DeepLumiere:main

DeepLumiere commented Jun 11, 2026

Uh oh!

tommasocerruti left a comment

Uh oh!

tommasocerruti Jul 3, 2026

Uh oh!

DeepLumiere Jul 3, 2026 •

edited

Loading

Uh oh!

tommasocerruti Jul 3, 2026

Uh oh!

DeepLumiere Jul 3, 2026

Uh oh!

tommasocerruti Jul 3, 2026

Uh oh!

DeepLumiere Jul 3, 2026

Uh oh!

tommasocerruti Jul 3, 2026

Uh oh!

DeepLumiere Jul 3, 2026

Uh oh!

tommasocerruti Jul 3, 2026

Uh oh!

tommasocerruti Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

DeepLumiere commented Jun 11, 2026

Uh oh!

tommasocerruti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DeepLumiere Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DeepLumiere Jul 3, 2026 •

edited

Loading