Skip to content

feat(onboarding-eval): add isolated multi-agent evaluation - #286

Draft
joaodinissf wants to merge 6 commits into
mainfrom
feat/onboarding-eval
Draft

feat(onboarding-eval): add isolated multi-agent evaluation#286
joaodinissf wants to merge 6 commits into
mainfrom
feat/onboarding-eval

Conversation

@joaodinissf

@joaodinissf joaodinissf commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds an isolated, manually dispatched onboarding evaluation that measures whether agents can onboard pinned public Maven repositories with the current Frontseat source tree.

The supported comparison modes are MCP plus skills, MCP only, skills only, and neither. The default run uses the smoke repository set and OpenRouter's moving DeepSeek V4 Flash alias (~deepseek/deepseek-v4-flash-latest); explicit repository and model overrides remain available.

Remote workflow

  • onboarding-eval.yml is dispatch-only: it has no push, pull-request, or schedule trigger.
  • Local provenance is the default and builds Frontseat plus its plugins from the workflow's checked-out commit before evaluation.
  • The reusable workflow owns the model matrix, isolated container execution, summaries, and retained artifacts.
  • Large matrices require explicit confirmation; repository parallelism and per-repository timeouts are bounded inputs.

Security and isolation

  • The workflow has read-only repository permissions and disables persisted checkout credentials.
  • Only the credential proxy receives OPENROUTER_API_KEY; the model container receives a constrained proxy URL instead of the key.
  • Model requests are restricted to the selected allowlisted model, accepted API paths, request count, and body size.
  • The model container is unprivileged, read-only, capability-free, and limited to a dedicated writable evaluation root.
  • The manifest contains only pinned public repositories. No private repository names, local paths, credentials, executables, or generated benchmark artifacts are committed.

Scoring and provenance

Reports record repository and Frontseat provenance, model, mode, concurrency, tool usage, cost when available, errors, and deterministic checks for onboarding, clean synchronization, tool specifications, conformance, and build success.

Review performed for this draft

  • Confirmed the branch is based directly on current origin/main after the repository history rewrite.
  • Reviewed the source-build and credential boundaries statically.
  • Checked shell syntax, whitespace, committed file types, and credential/private-data patterns.
  • Verified the default model alias against OpenRouter's live model catalog.

Remote validation

  • Repository CI passed at rewritten head 0c5ad08, including frontseat build //.
  • Source-build smoke evaluation completed successfully at the same head with DeepSeek V4 Flash Latest, both MCP and skills enabled, and local provenance.
  • Both pinned repositories onboarded and synchronized successfully. The measured result was 60% mean compliance (0/2 all-green): Gson failed conformance and both Maven builds failed during asset resolution; Commons Lang also produced an invalid tool specification.

The end-to-end benchmark was intentionally executed in GitHub Actions rather than consuming local laptop resources.

@joaodinissf
joaodinissf force-pushed the feat/onboarding-eval branch from bebc332 to 0c5ad08 Compare August 2, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant