fix(agy): register work_dir as an agy workspace via --add-dir - #585
Merged
Conversation
Setting the subprocess cwd made agy resolve reads against work_dir, but its shell and write tools still ran in <appDataDir>/scratch, so agent-created files landed outside the scenario directory. agy tracks the read workspace and the tool execution directory separately; only --add-dir sets the latter. The flag is passed only when a scenario declares work_dir, leaving the MCP startup probe and work_dir-less scenarios unchanged.
test_run_command_argv_shape already asserts exact argv equality for a command with no work_dir, so it fails if the flag is emitted unconditionally.
omkargaikwad23
requested review from
IsmailMehdi,
helloeve and
prernakakkar-google
as code owners
August 31, 2026 05:19
…symmetry Route the test through create_command so the full evaluator path is pinned: work_dir -> CLICommand.cwd -> --add-dir plus subprocess cwd. Document why the fake_home cwd fallback is not mirrored into --add-dir.
Collaborator
Author
|
/gcbrun |
Collaborator
|
Please correct/rephrase the description. |
Collaborator
|
/gcbrun |
prernakakkar-google
approved these changes
Aug 31, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
work_dirwas only passed to agy as the subprocess cwd. agy uses the cwdfor file reads, but runs its shell and write tools in
<appDataDir>/scratch,so files the agent created never landed in
work_dir. Only--add-dirsetsthe tool directory.
This passes
--add-dir <work_dir>in addition to the cwd, emitted only whena scenario declares
work_dir— the MCP startup probe and existing scenariosare unaffected. The
fake_homecwd fallback is not mirrored into the flag,since it holds harness internals rather than scenario data.
Only agy needs this.
claude_code,codex_cliandgemini_clihonour theprocess cwd directly.
Test plan
pytest evalbench/test/agy_cli_test.py— 68 passedcreate_command(cwd=...)→--add-dir→ subprocess cwdwork_dircase is covered by existing exact-argv tests, which fail if the flag isalways emitted
work_dirset: the agent's file landed inwork_dirand scratch was empty(before the fix, the reverse)
work_dirThe rubric scorer only reads the transcript, so the scenario scored 100%
before and after. The fix was verified on disk.