Repository navigation
fix(examples): make the coding-agent harness self-correcting, add tests - #1
Closed
haigou-web wants to merge 2 commits into
Closed
haigou-web wants to merge 2 commits into
haigou-web wants to merge 2 commits into
Conversation
Address the review findings on the coding-agent example: - Drop the unused imports in tools.py; `requests` was not in requirements.txt, so a clean install failed to import at all. - Route tool actions back through create_prompt, so the prompt-shaping seam runs before every model call rather than only the first. - Report an unknown tool name back to the model as a tool error instead of leaving call_llm with no matching transition. - Catch malformed tool-call JSON in OpenAIClient and hand it to the model as an error rather than raising. - Wrap tool execution in try/except so a raising tool becomes an error result the model can react to. - Create the workspace before running a command, so a model that starts with run_bash does not hit FileNotFoundError. - Resolve paths with realpath so a symlink inside the workspace cannot be followed out of it. - Skip the graphviz render when the binary is unavailable, instead of crashing before the example runs. - Add the ASF license header to README.md.
Cover the scripted happy path, step-budget termination, unknown tools, malformed tool arguments, tool failures, and workspace confinement for reads and writes. Driven by ScriptedClient, so no API key is needed, and tools.WORKSPACE is patched to tmp_path so tests never touch a real ./workspace.
Author
|
Closing this one: it was a stacked PR against a fork branch while the upstream example was still under review. The work has since gone in directly against Nothing here needs to be actioned on your side — thanks for the branch, it is what made the upstream submission possible. |
Owner
|
@haigou-web thanks for offering to help with these — I’ve worked through them directly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to apache#886, addressing the findings from @DeepGandhi01's review and the two requirements @skrawcz listed. Each item below is reproducible with
ScriptedClientand no API key.Review findings
tools.py(requests,openai,inspect,json,typing,burr.core, and a duplicateos).requestswas not inrequirements.txt, so a clean install failed to import at all.create_promptonly ran once — tool actions now transition back tocreate_promptinstead of straight tocall_llm, so the prompt-shaping seam runs before every model call as the README describes.call_llmdetects a tool that isn't registered, appends a{"error": "unknown tool: ..."}tool message, and loops back so the model can retry.execute_toolwraps the call and returns{"error": "TypeError: ..."}as the tool result.run_bashalso creates the workspace first, so a model that starts with a command no longer hitsFileNotFoundError.OpenAIClient—JSONDecodeErroris caught and surfaced to the model as an error, same path as (4)._resolvecould be escaped via symlinks — now usesrealpathfor bothrootandfull, so a link inside the workspace can't be followed out of it.app.visualize()is wrapped intry/except, so a first run without the binary prints a one-line note instead of a traceback.Maintainer requirements
README.md.test_coding_agent.pywith 10 tests covering the scripted happy path, step-budget termination, unknown tools, malformed tool arguments, tool failures, and workspace confinement for both reads and writes (../,sub/../../,/etc/passwd). Driven byScriptedClient, so no API key is needed;tools.WORKSPACEis patched totmp_pathso tests never touch a real./workspace.Verified
pytest test_coding_agent.py→ 10 passedpytest validate_examples.py(the CI gate) → 48 passedpython application.pywith no API key →Created hello.py and confirmed it runs.W292missing-trailing-newline fixes)Note: the base of this PR is the apache#886 branch rather than
main, so the diff here is only the follow-up — the example itself is unchanged from apache#886.One thing I left alone: the default workspace is
./workspace, so running the example drops a directory inside the repo. CI doesn't executeapplication.pyso nothing breaks, but a.gitignoreline may be worth adding — happy to include it if you want it in this PR.