Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
f872ce5
Update source URL and modify log file structure
DeepLumiere Jun 10, 2026
b92fab5
Fix UUID handling and improve log writing format
DeepLumiere Jun 10, 2026
83007f4
added aggregate and instance level jsonl
DeepLumiere Jun 11, 2026
fa48968
Refactor log writing to use UUID filename segments
DeepLumiere Jun 11, 2026
419d029
Merge branch 'evaleval:main' into main
DeepLumiere Jun 18, 2026
0b09425
Add script to run and manage Every Eval Ever adapters
DeepLumiere Jun 19, 2026
42618d6
Refactor run_adapters.py for repo ID and upload changes
DeepLumiere Jun 20, 2026
162f85d
First Run of Adapter Cron
DeepLumiere Jun 20, 2026
2e2b85c
Add environment to adapter cronjob workflow
DeepLumiere Jun 20, 2026
ed82ecb
Enhance filename sanitization in io.py
DeepLumiere Jun 21, 2026
9cbffab
Fix upload function to use upload_folder
DeepLumiere Jun 21, 2026
6dbe1af
Change cron schedule to run weekly on Sundays
DeepLumiere Jun 30, 2026
2595b89
Update repository ID and add revision parameter
DeepLumiere Jun 30, 2026
0482799
Merge pull request #1 from evaleval/main
DeepLumiere Jun 30, 2026
7d38126
Update run_adapters.py
DeepLumiere Jun 30, 2026
5c025ce
Update run_adapters.py
DeepLumiere Jun 30, 2026
1a7ba4c
Update run_adapters.py
DeepLumiere Jun 30, 2026
c8727c6
Update argument parser description in run_adapters.py
DeepLumiere Jun 30, 2026
d390cc2
Add files via upload
DeepLumiere Jul 1, 2026
fc0f861
Update run_adapters.py
DeepLumiere Jul 1, 2026
3005137
Update run_adapters.py
DeepLumiere Jul 2, 2026
7d65667
Modify get_dynamic_args to accept environment variables
DeepLumiere Jul 2, 2026
9744e7e
Update run_adapters.py
DeepLumiere Jul 2, 2026
10229ab
Refactor adapter staleness logic and input parsing
DeepLumiere Jul 2, 2026
ecc370b
Update run_adapters.py
DeepLumiere Jul 2, 2026
a95103e
Update run_adapters.py
DeepLumiere Jul 2, 2026
abcb627
Update run_adapters.py
DeepLumiere Jul 2, 2026
f4fec94
Refactor run_adapters.py for clarity and efficiency
DeepLumiere Jul 2, 2026
3ba7388
feat: add cron job and script for running adapters
Dudecomputerscience Jul 4, 2026
21f59de
Merge remote-tracking branch 'origin/main'
Dudecomputerscience Jul 4, 2026
d49c8df
feat: add configuration files and scripts for data processing
Dudecomputerscience Jul 4, 2026
a8b7ed1
Update adapter_cron.yml
DeepLumiere Jul 4, 2026
b47f6fa
Delete .github/workflows/adapter_cronjob.yml
DeepLumiere Jul 6, 2026
b868b7e
Refactor run_adapters.py for clarity and efficiency
DeepLumiere Jul 6, 2026
b4fa9f8
Update adapter.py
DeepLumiere Jul 6, 2026
a77d15f
Change import path for eval_types module
DeepLumiere Jul 6, 2026
2ffe902
Increase thresholds for heavy adapters classification
DeepLumiere Jul 6, 2026
0c409bb
Increase HEAVY_TIME_S threshold from 240 to 900
DeepLumiere Jul 7, 2026
84f8056
add output directory argument to adapter scripts and update dependencies
Dudecomputerscience Jul 7, 2026
01d1834
feat: add input option to force run all adapters in cron job
Dudecomputerscience Jul 7, 2026
657a3a7
Update cron job description for force_all input
DeepLumiere Jul 7, 2026
05aac8a
Fixed dataset link
DeepLumiere Jul 7, 2026
38bf464
Upgrade to run_adapters.py
Dudecomputerscience Jul 7, 2026
0eeeb7d
Merge remote-tracking branch 'origin/main'
Dudecomputerscience Jul 7, 2026
9ea6216
Fork Update
DeepLumiere Jul 8, 2026
9c84a92
Upgrade to run_adapters.py
Dudecomputerscience Jul 8, 2026
f70b24f
Merge remote-tracking branch 'origin/main'
Dudecomputerscience Jul 8, 2026
5089eae
Increased limits
DeepLumiere Jul 8, 2026
a06ab07
Upgrade to run_adapters.py
Dudecomputerscience Jul 9, 2026
c57bfe1
Merge remote-tracking branch 'origin/main'
Dudecomputerscience Jul 9, 2026
cf36ec8
Added Mercor See Crate
DeepLumiere Jul 10, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# EEE Platform environment variables
# Copy this to .env and fill in your values

# --- Database ---
DATABASE_URL=postgresql+asyncpg://eee:eee_password@localhost:5432/eee_platform
DB_ECHO=false

# --- Google OAuth ---
# Get these from https://console.cloud.google.com/apis/credentials
GOOGLE_CLIENT_ID=your-google-client-id
GOOGLE_CLIENT_SECRET=your-google-client-secret
GOOGLE_REDIRECT_URI=http://localhost:8000/api/v1/auth/google/callback

# --- JWT ---
SECRET_KEY=change-this-to-a-random-string-in-production

# --- General ---
DEBUG=true
ALLOWED_ORIGINS=["http://localhost:3000","http://localhost:5173","http://localhost:8000"]
50 changes: 50 additions & 0 deletions .github/workflows/adapter_cron.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Run Adapters Cron

on:
schedule:
- cron: '0 0 * * *' # Daily at midnight UTC
workflow_dispatch: # Allow manual triggering
inputs:
force_all:
description: 'Force run all adapters'
required: false
type: boolean
default: false

jobs:
run-adapters:
runs-on: ubuntu-latest
environment: adapter_cron
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: "pyproject.toml"

- name: Install dependencies
run: uv sync --locked --all-extras

- name: Run Adapters Orchestrator
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
MERCOR_EVAL_API_EVALEVAL_KEY: ${{ secrets.MERCOR_EVAL_API_EVALEVAL_KEY }}
ARTIFICIAL_ANALYSIS_API_KEY: ${{ secrets.ARTIFICIAL_ANALYSIS_API_KEY }}
KAGGLE_USERNAME: ${{ secrets.KAGGLE_USERNAME }}
KAGGLE_KEY: ${{ secrets.KAGGLE_KEY }}
LLM_STATS_API_KEY: ${{ secrets.LLM_STATS_API_KEY }}
run: |
if [ "${{ github.event.inputs.force_all }}" == "true" ]; then
uv run python utils/scripts/run_adapters.py --force-all
else
uv run python utils/scripts/run_adapters.py
fi
10 changes: 10 additions & 0 deletions .idea/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 17 additions & 0 deletions .idea/every_eval_ever.iml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 14 additions & 0 deletions .idea/inspectionProfiles/Project_Default.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/inspectionProfiles/profiles_settings.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions .idea/misc.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions .idea/modules.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading