Skip to content

Add a langgraph-node runner for regression-testing LangGraph agents #36

Description

@hoainho

Context

eval-harness ships with one runner (opencode-skill). The runner contract is documented in docs/runners.md — 4 subcommands (prepare, spawn, fingerprint, teardown), ~150 lines of bash for a minimum viable runner.

LangGraph is the most-asked-about target. This issue scopes the work.

What to build

scripts/eval/runners/langgraph-node.sh implementing the runner contract:

  • prepare: create an ephemeral Python venv, pip install -r case-requirements.txt (if present in the case fixture), import-check the target module
  • spawn: invoke the LangGraph graph entry node with the case prompt, capture transcript via langgraph.utils.tracer or LangSmith local-export. Write transcript JSONL to $transcript_out.
  • fingerprint: hash the graph definition module + prompt templates + any @tool-decorated functions. Output: hex SHA to stdout.
  • teardown: clean up the venv (or leave for caching, configurable).

Plus:

  • 3 example cases in examples/langgraph-runner/ (or similar) demonstrating shell, jq_path_contains, and llm_judge checks against a small LangGraph graph
  • Test under scripts/eval/tests/runner_langgraph.sh using a stubbed minimal graph (no real LLM call)
  • Update docs/runners.md to move LangGraph from "Roadmap" to "Shipped runners"

Acceptance criteria

  • Runner implements all 4 subcommands per docs/runners.md
  • kind: shell, kind: jq_path_contains, kind: output_contains all work against the runner
  • attribute.sh correctly attributes SKILL_CHANGED vs FIXTURE_STALE vs MODEL_CHANGED for LangGraph cases (the fingerprint subcommand drives this)
  • Stubbed test runs in CI without an Anthropic key
  • Example fixtures fit in < 50 lines of Python

Why this is help-wanted not good-first-issue

  • Touches a new framework (LangGraph), so good Python familiarity needed
  • Involves designing the venv-caching strategy (small but real engineering decision)
  • Cross-cuts the harness manifest layer (need to add graph_sha next to skill_sha)

If you've shipped a LangGraph agent and want regression testing on it: this PR is the highest-leverage contribution you can make to eval-harness right now. Ping in discussion #28 to coordinate.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is neededrunnerAdds or modifies a runner adapter

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions