feat: pluggable scientific workspace, canonical resource policy, multi-task server - #205
Merged
Conversation
- Rename server/ to revocompute, module pssm_gremlin.py → app.py - Class GremlinConfig → ComputeConfig, URL prefix /PSSM_GREMLIN/ → /compute/ - Legacy 302 redirect from /PSSM_GREMLIN/ → /compute/dashboard - Template pssm_gremlin_dashboard.html → dashboard.html - Docker images: revodesign-pssm-gremlin-* → revodesign-revocompute-* - Container paths: /app/pssm_gremlin/ → /app/revocompute/ - Celery app: pssm_gremlin_server → revocompute - Update docs, CI config, nginx config, docker-compose, env files - Fix all test fixtures and sys.modules patching Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lin/ Move runner-specific files into a per-task-type directory: - docker/runner/Dockerfile → docker/runners/pssm_gremlin/Dockerfile - run_gremlin.sh → docker/runners/pssm_gremlin/run.sh - scripts/ → docker/runners/pssm_gremlin/scripts/ - env/GREMLIN.yml → docker/runners/pssm_gremlin/GREMLIN.yml Update docker-compose.yml, task_types.yaml, app.py fallback, README, and test paths accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- DB: add task_type, input_form, uploaded_files columns to tasks_table - API: accept task_type in POST /compute/api/post, add GET /compute/api/types - API: generic _validate_input_upload replaces _validate_fasta_upload - API: task_type included in dashboard task status responses - Frontend: create_task page gains task type selector, dynamic file accept, params form, generic labels - Frontend: dashboard removes hardcoded GREMLIN trace fallback/labels, adds task type badge - Branding: REvoDesign GREMLIN → REvoCompute across all templates, JS, CSS - Tests: 273 passing Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t override, and docs - task_runtime: auto-load registry from CONFIG_DIR, built-in gremlin fallback - schemas: TaskSubmissionRequest with registry-aware param validation - routes: GET /compute/api/types/<name> endpoint, flat params[key]=value parsing - task_runtime: entrypoint override with CLI args (-i/-o/-r), non-root user - docker-compose: CONFIG_DIR env var for config deployment - Dockerfile: tolerant groupadd/useradd, simplified entrypoint (no ldconfig/runuser) - runner YAML: local dev settings with per-type mounts, env, and resource limits - docs: multi-task architecture in server README and dev-guide/server.md - changelog: multi-task entries under [Unreleased] Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mount the input directory at /workspace/inputs instead of the single file, so the runner sees /workspace/inputs/<original_name>.fasta and derives the correct instance name (e.g., 2KL8 instead of inputs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…_files
- schemas: InputEntity and InputForm pydantic models for typed input tracking
- schemas: TaskSubmissionRequest.coerce_params() for type-aware value coercion
- routes: build entity list from submission + file metadata at upload time
- routes: input_form stores {user, submitted_at, entities: [...]}
- task_runtime: _run_in_docker reads entities — file entities give mounts/-i,
param entities give TASK_PARAMS and CLI flags
- task_runtime: _execute_compute_task reads entities from DB, not Celery args
- db: drop obsolete uploaded_files column
- tests: update _insert_pending_task and _fake_runner to use entities
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Fix datetime.timezone import error (use timezone.utc not datetime.timezone.utc) - Mount upload/ directory for Docker input instead of results/<md5sum>/ which may not sync to the host filesystem in time for Docker daemon - Use hardlink in upload/ dir to preserve original filename in container - Clean up hardlink after container exits - Update entity stored_at to point to upload_path Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add revocompute/job/ package with Job ABC and JobState enum - Extract DockerJob from _run_in_docker() into job/runners/docker_runner.py - Add SlurmJob with sbatch script generation, squeue polling, scancel - Fix .upload suffix: sbatch script hardlinks to original extension - Add runner/container_runtime/slurm_image fields to RunnerConfig - Unified dispatch via _create_job() factory in task_runtime.py - Add Apptainer .def files under docker/runners/<name>/ - Lazy Docker client init: avoids connecting at import time Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Parse --use-slurm, --allowed-slurm-queue, --build-sif flags - validate_slurm_images(): check SIF files exist, print build commands if missing - build_slurm_images(): auto-build .sif from docker/runners/<name>/*.def - Wire into cmd_restart() before docker compose up Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dd runner tests - Move _RUNNER_STAGE_PREFIX and extract_stage_from_log_line into job/_stages.py, imported by both docker_runner and task_runtime. - Delete _run_in_docker() thin wrapper — all paths use _run_compute_job(). - Fix 3 task tests that were monkeypatching the (now-dead) _run_in_docker; they now patch _run_compute_job. - Add 40 unit tests: test_job_abc.py (ABC completeness), test_docker_runner.py (mocked Docker submit/poll/cancel), test_slurm_runner.py (sbatch generation, state mapping, subprocess mocking). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add COMPOSE_PROJECT_NAME + unique image tags to isolate test from prod - Add docker-compose.slurm.yml override: SLURM tools, MUNGE, CONFIG_DIR mounts, host networking for worker and web - restart.sh: compose_files() helper for --use-slurm override merging, pssm_ prefix fallback for .def file discovery, move SIF build after Docker build so the base image exists first, read CONFIG_DIR for deployed runner configs instead of repo source - Seed task_type_config rows in manage DB from registry on startup - Add slurm_job_id column to tasks table for scancel persistence - Call scancel on task cancel when slurm_job_id is present - Add Apptainer .def files using docker-daemon bootstrap (local images) Co-Authored-By: Claude <noreply@anthropic.com>
- Add allow_gpu_use column to users table (default false) - AdminUpdateUserRequest schema accepts allow_gpu_use flag - Admin user control page: GPU badge in view row, checkbox toggle in edit row - Task submission: if task type has gpus=true and user lacks allow_gpu_use, return 403 before enqueueing Co-Authored-By: Claude <noreply@anthropic.com>
Replace sbatch + squeue/sacct polling with srun + subprocess.Popen. Stdout/stderr captured directly via pipes in background threads, same living-output pattern as Docker runner. No sacct dependency, no #SBATCH directive ordering bugs, no heuristic completion detection. srun flags are built from the same slurm_config DB fields; the wrapper script handles input staging, APPTAINERENV_ exports, and apptainer run. cancel() terminates the srun process and falls back to scancel. Co-Authored-By: Claude <noreply@anthropic.com>
srun runs on the host (via host networking) and cannot read files from the container /tmp. Write the wrapper script into output_dir which is a host-mounted path. Also: drop unused tempfile import, defer script cleanup (srun needs to read the script).
Add comprehensive SLURM documentation to both docs/dev-guide/server.md and server/README.md: - Architecture: SLURM runner backend design decision, job/ module layout - Runner contract: SLURM + Apptainer runner details (wrapper script, srun+Popen, live stage parsing, APPTAINERENV_ exports) - Deployment: SLURM setup steps, docker-compose.slurm.yml, SIF build, compose project isolation, GPU privilege gating - Env vars: SLURM_ENABLED, CONFIG_DIR, COMPOSE_PROJECT_NAME - restart.sh flags: --use-slurm, --build-sif Co-Authored-By: Claude <noreply@anthropic.com>
Strip over-engineering found by CLAUDE.md principles audit: delete: _parse_job_id_from_stderr (120s timeout polling) delete: _SRUN_JOB_ID_RE regex + import re delete: dual env export loops — APPTAINERENV_ alone works delete: dead env_flag_parts list delete: scancel fallback in cancel() — never works with synthetic IDs delete: import time, import re, from pathlib shrink: _render_env merged into _render_apptainer_invocation shrink: _build_wrapper_script returns str, not Path Net: -89 lines (-25%). Same contract, same output.
Remove tests for removed symbols (_SQUEUE_STATE_MAP, _render_script, scancel) and replace with tests for the current srun/Popen-based implementation. Add _sanitize_name tests for SLURM job name validation. Co-Authored-By: Claude <noreply@anthropic.com>
… dedup Critical: fix broken job cancellation (capture real SLURM_JOB_ID, gate slurm_job_id persistence to SlurmJob only), remove process-global SIGINT handler from DockerJob.poll(), add try/except to param coercion preventing 500 on bad values, HTML-escape user data in email bodies + username charset validation, key rate limiter on X-Forwarded-For matching _client_ip(). High: unique-per-job hardlink name + FileExistsError guard, remove wrapper script from output_dir after poll(), revoke sessions on password reset, remove duplicate registry loading from app.py, guard stage callbacks in both runners, cap stderr_lines at 200 in DockerJob.poll(). Delete: InputEntity/InputForm schemas, dead ManageDatabase methods, duplicate verify-email endpoint, duplicate _task_delete_allowed, All Keys tab (~140 JS lines), NPROC env var (never read), dead currentUser variable. Cleanup: theme toggle on configuration page, GPU column in user_control.html, escapeHtml consolidated into auth-api.js, sub-tabs CSS deduplicated into base.css, REvoDesign GREMLIN renamed to REvoCompute everywhere, test uploads use tmp_path, unused test imports removed, tampered-token test renamed, tests/conftest.py import order restored with # isort:off guard + assert. Co-Authored-By: Claude <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedToo many files! This PR contains 177 files, which is 77 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (3)
📒 Files selected for processing (177)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
 |
|
Overall Grade |
Security Reliability  Complexity Hygiene  |
Code Review Summary
| Analyzer | Status | Updated (UTC) | Details |
|---|---|---|---|
| Python |  |
Aug 12, 2026 3:46p.m. | Review ↗ |
Important
AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| UnusedCode | 4 medium |
| BestPractice | 2 medium |
| ErrorProne | 24 high |
| Security | 7 medium 61 high |
| CodeStyle | 2 minor |
🟢 Metrics 3059 complexity · 79 duplication
Metric Results Complexity 3059 Duplication 79
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
- py2Dmol renders with its own default color (auto/rainbow) on mount; sync to activeColorMode (pLDDT) after initialization completes. - Show the actual Mol* error message instead of a generic string so the root cause of CDN/API failures is visible. Co-Authored-By: Claude <noreply@anthropic.com>
Passes ENABLED_TASKRUNNERS env var to docker-compose, gating which task runners are loaded. Accepts both --enabled-runners=<csv> and --enabled-runners <csv> forms. Default: all registered runners. Co-Authored-By: Claude <noreply@anthropic.com>
The old server/docker/runner/Dockerfile was removed; point CI and compat tests at the GREMLIN runner Dockerfile under the new server/docker/runners/pssm_gremlin/ layout. Co-Authored-By: Claude <noreply@anthropic.com>
Logout now requires Bearer auth (CSRF hardening); the test was using cookie-only auth and getting 403 instead of invalidating the token. Co-Authored-By: Claude <noreply@anthropic.com>
When ENABLED_TASKRUNNERS is set, only the matching runner Docker images are built and validated. Empty (default) builds all runners. Co-Authored-By: Claude <noreply@anthropic.com>
…ulls All runtime_manifest consumers now respect ENABLED_TASKRUNNERS: Docker build, SIF validation, SIF build, prepared-image validation, and prod docker pull. Only the structural config read is unfiltered. Co-Authored-By: Claude <noreply@anthropic.com>
The config directory defaulted to <SERVER_DIR>/../config inside the container, which didn't exist. Copy the real config into the test workspace at the expected relative path. Co-Authored-By: Claude <noreply@anthropic.com>
The test builds only the GREMLIN image; limiting ENABLED_TASKRUNNERS to gremlin ensures startup validation only checks that image. Co-Authored-By: Claude <noreply@anthropic.com>
config_dir defaulted to <SERVER_DIR>/../config, which is outside the mounted volume. Copy into SERVER_DIR and set CONFIG_DIR explicitly. Co-Authored-By: Claude <noreply@anthropic.com>
The runner config has hardcoded host_paths for database mounts that only exist on the developer's machine. sed them to the CI test workspace paths before launching the stack. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
GHA Docker daemon may not access /tmp paths. Move databases under WORK_DIR/state/ which is inside the mounted server directory tree. Co-Authored-By: Claude <noreply@anthropic.com>
The docker runner checks os.path.exists(host_path) inside the worker container, which only sees mounted volumes. SERVER_DIR is mounted; miniuc must live inside it for the check and the bind mount to work. Co-Authored-By: Claude <noreply@anthropic.com>
esmfold_inference.py imports openfold.np.residue_constants via esm2.esmfold.v1.misc, but the [esmfold] extra does not pull openfold. Co-Authored-By: Claude <noreply@anthropic.com>
The test builds a randomly-tagged runner image; task_types.yaml still points at the generic name. Sed it to the test image. Co-Authored-By: Claude <noreply@anthropic.com>
restart.sh validation compares task_types.yaml docker_image and the .def From: line against the expected image name + :latest. Tag the test-built runner image accordingly instead of sed-ing config files. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Docker container IDs persisted at submit time need a declared column; SQLAlchemy raised Unconsumed column names otherwise. Backfill migration for existing task tables. Co-Authored-By: Claude <noreply@anthropic.com>
openfold 1.0.1 requires Python >=3.10 (only 0.0.1 supports 3.9); the image was pinned to python3.9/torch1.12. Move to Python 3.12 via deadsnakes PPA, torch 2.1.2+cu121, and CUDA 12.1 runtime. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
PyPI 'openfold' has only a 0.0.1 placeholder release. The real package installs from the aqlaboratory/openfold GitHub repo; v2.2.0 is the torch-2.x / Python-3.12 compatible tag. It has no install_requires, so dm-tree and biopython (its runtime deps for the esmfold import chain) are pinned explicitly. Co-Authored-By: Claude <noreply@anthropic.com>
…version lists, nvcc) Co-Authored-By: Claude <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #205 +/- ##
=======================================
Coverage 74.09% 74.10%
=======================================
Files 122 122
Lines 15575 15575
=======================================
+ Hits 11541 11542 +1
+ Misses 4034 4033 -1
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactors the REvoCompute server from a single-task PSSM-GREMLIN service into a multi-task scientific compute platform. 150 commits.
Pluggable scientific input workspace
InputCapabilityschema in task registry — files, sequence, structure, regions, parameters, reviewPluginHost/PluginRegistryfor lifecycle-safe mounting, collection, validation, and teardowninput_workspaceget a derived workspace from accepted extensions and paramsCanonical resource policy
resource_policy.py: typedResolvedResourceswith singleresolve_resources()pathcpus/memoryreplacing ad-hocnproc/maxmem/slurm_*sprawlgpu:1; CPU tasks cannot inherit GPU GRESRunner integration
nano_cpus/mem_limit+ env vars,srunargsdocker stop/scancelfrom DB handlessacctMulti-task support
Result workspace
ResultPreviewHost— text, table, image, structure, MSAFrontend improvements
/api/types/<name>/helpendpoint)Security hardening
token_version+ nonce (no replay within 1-hour window)AUTH_SECRET_KEYenv override for stable signing keys across restartstoken_versionbumpSecureflag usesrequest.is_secure(actual TLS, not config string)chmod 600on users.sqlite3Auth fix
authFetch(callsensureToken) so cookie-only sessions workTests
Legacy cleanup
nproc/maxmem/slurm_cpus_per_task/slurm_memmigration scaffolding/PSSM_GREMLIN/dashboard→/compute/dashboard302 redirect🤖 Generated with Claude Code