Skip to content

fix(examples): make the coding-agent harness self-correcting, add tests - #1

Closed
haigou-web wants to merge 2 commits into
Agentic-JJ-Web3:docs/coding-agent-harness-examplefrom
haigou-web:fix/coding-agent-harness
Closed

haigou-web wants to merge 2 commits into
Agentic-JJ-Web3:docs/coding-agent-harness-examplefrom
haigou-web:fix/coding-agent-harness

Conversation

@haigou-web

Copy link
Copy Markdown

Follow-up to apache#886, addressing the findings from @DeepGandhi01's review and the two requirements @skrawcz listed. Each item below is reproducible with ScriptedClient and no API key.

Review findings

  1. Clean-install imports — dropped the unused imports in tools.py (requests, openai, inspect, json, typing, burr.core, and a duplicate os). requests was not in requirements.txt, so a clean install failed to import at all.
  2. create_prompt only ran once — tool actions now transition back to create_prompt instead of straight to call_llm, so the prompt-shaping seam runs before every model call as the README describes.
  3. Unknown tool name halted the run — call_llm detects a tool that isn't registered, appends a {"error": "unknown tool: ..."} tool message, and loops back so the model can retry.
  4. Tool errors crashed the run — execute_tool wraps the call and returns {"error": "TypeError: ..."} as the tool result. run_bash also creates the workspace first, so a model that starts with a command no longer hits FileNotFoundError.
  5. Malformed tool-call JSON crashed OpenAIClient — JSONDecodeError is caught and surfaced to the model as an error, same path as (4).
  6. _resolve could be escaped via symlinks — now uses realpath for both root and full, so a link inside the workspace can't be followed out of it.
  7. Graphviz nit — app.visualize() is wrapped in try/except, so a first run without the binary prints a one-line note instead of a traceback.

Maintainer requirements

  • ASF license header added to README.md.
  • Automated tests — new test_coding_agent.py with 10 tests covering the scripted happy path, step-budget termination, unknown tools, malformed tool arguments, tool failures, and workspace confinement for both reads and writes (../, sub/../../, /etc/passwd). Driven by ScriptedClient, so no API key is needed; tools.WORKSPACE is patched to tmp_path so tests never touch a real ./workspace.

Verified

  • pytest test_coding_agent.py → 10 passed
  • pytest validate_examples.py (the CI gate) → 48 passed
  • python application.py with no API key → Created hello.py and confirmed it runs.
  • black 23.11.0 / isort 5.12.0 / flake8 6.1.0 clean (including two W292 missing-trailing-newline fixes)

Note: the base of this PR is the apache#886 branch rather than main, so the diff here is only the follow-up — the example itself is unchanged from apache#886.

One thing I left alone: the default workspace is ./workspace, so running the example drops a directory inside the repo. CI doesn't execute application.py so nothing breaks, but a .gitignore line may be worth adding — happy to include it if you want it in this PR.

Address the review findings on the coding-agent example:

- Drop the unused imports in tools.py; `requests` was not in
  requirements.txt, so a clean install failed to import at all.
- Route tool actions back through create_prompt, so the prompt-shaping
  seam runs before every model call rather than only the first.
- Report an unknown tool name back to the model as a tool error instead
  of leaving call_llm with no matching transition.
- Catch malformed tool-call JSON in OpenAIClient and hand it to the model
  as an error rather than raising.
- Wrap tool execution in try/except so a raising tool becomes an error
  result the model can react to.
- Create the workspace before running a command, so a model that starts
  with run_bash does not hit FileNotFoundError.
- Resolve paths with realpath so a symlink inside the workspace cannot be
  followed out of it.
- Skip the graphviz render when the binary is unavailable, instead of
  crashing before the example runs.
- Add the ASF license header to README.md.
Cover the scripted happy path, step-budget termination, unknown tools,
malformed tool arguments, tool failures, and workspace confinement for
reads and writes. Driven by ScriptedClient, so no API key is needed, and
tools.WORKSPACE is patched to tmp_path so tests never touch a real
./workspace.
@haigou-web

Copy link
Copy Markdown
Author

Closing this one: it was a stacked PR against a fork branch while the upstream example was still under review. The work has since gone in directly against main as apache#951, which carries the same fixes plus two more that came out of review.

Nothing here needs to be actioned on your side — thanks for the branch, it is what made the upstream submission possible.

@haigou-web haigou-web closed this Oct 6, 2026
@Agentic-JJ-Web3

Copy link
Copy Markdown
Owner

@haigou-web thanks for offering to help with these — I’ve worked through them directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants