Add Harbor mode: run SIA self-improvement on external benchmarks#21
Open
prannayHexo wants to merge 1 commit into
Open
Add Harbor mode: run SIA self-improvement on external benchmarks#21prannayHexo wants to merge 1 commit into
prannayHexo wants to merge 1 commit into
Conversation
Opt-in via --harbor_dataset / --harbor_task_dir. Each generation's target_agent.py is run inside the benchmark's Docker containers and scored by Harbor's native verifiers; per-task rewards + trajectories feed the existing feedback loop. Local behavior and the golden-master prompts are unchanged (Harbor text is injected only when the flag is set). - sia/harbor_agent.py: BaseAgent adapter that runs the generated agent in-container - sia/harbor_runner.py: drives the harbor CLI, folds results into results.json + trajectories - sia/tasks/_shared/reference_harbor_agent.py: in-container reference template - cli/prompts/run_setup/layout/orchestrator: flags, prompt injection, routing - docs/harbor.md, README, pyproject [harbor] extra, tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #20 (closed; could not be reopened after a force-push). Same change, with
reference_harbor_agent.pyrestyled to the terse/no-comment house rules.What
Opt-in Harbor mode: SIA self-improves against external benchmarks from the Harbor registry (Terminal-Bench, SWE-Bench, AIME, …), scored by Harbor's own verifiers. Each generation's
target_agent.pyruns inside each task's Docker container; per-task rewards + trajectories feed the existing feedback loop.How
sia/harbor_agent.py—SIATargetAgent(BaseAgent): uploads the agent into each container, runs it (--working_dir / --instruction_file / --log_dir), downloads the trajectory, verifier scores.sia/harbor_runner.py— drives theharborCLI as a subprocess; parsesresult.json→results.json+ per-task trajectories.sia/tasks/_shared/reference_harbor_agent.py— in-container reference template.cli.py, snapshot-safe injected prompt block inprompts.py(base prompts untouched), harbor task-files + venv skip inrun_setup.py, routing inorchestrator.py.Usage
Validation
Four full
--max_gen 2runs (hello-world, log-summary, the realharbor downloadregistry path, and a CWEfix-code-vulnerabilitytask), each adversarially verified on disk — all scored a verifier-adjudicated 1.0 with the correct in-container contract. Regression: 124 passed / 0 failed, ruff clean, golden prompt snapshots intact.See docs/harbor.md.