Skip to content

docs: restructure README and add harness-eval references - #77

Merged
Aliciapet11 merged 3 commits into
mainfrom
docs/restructure-readme
Aug 17, 2026
Merged

docs: restructure README and add harness-eval references#77
Aliciapet11 merged 3 commits into
mainfrom
docs/restructure-readme

Conversation

@Benkapner

Copy link
Copy Markdown
Collaborator

Summary

The README was 691 lines long, mixing quick-start content with full reference documentation. This PR slims it down to 139 lines and moves the detailed sections into dedicated files under Docs/.

What moved

Section New location
Gates Architecture (types, modes, schema, existing gates, scorecard, policy config) Docs/gates-architecture.md
Submission Formats (Harbor, ASE, MCP, A2A, AEH directory layouts) Docs/submission-formats.md
Extensibility (adding engines, gates, gate categories with code examples) Docs/extensibility.md
Persistence (MinIO layout, PostgreSQL) Docs/persistence.md

Compass Facts Integration was already in Docs/compass_facts_integration.md.

What's new

Added harness-eval references:

  • In "How It Works" stage 2 (Test): listed as the third scanning tool alongside Cisco and the LLM quality review
  • In "Related Repositories" table: linked with description (27 rule categories, 97 rules)
  • In Docs/gates-architecture.md: documented both security scanners (Cisco + harness-eval) and both quality sources (LLM review + harness-eval)

What stayed in README

  • Intro, pipelines table, how-it-works overview (6 stages, concise)
  • Configuration example (metadata.yaml)
  • Repository structure
  • Related repositories (now includes harness-eval)
  • LLM access, prerequisites, documentation links, license

Moved gates architecture, submission formats, extensibility, and
persistence sections into dedicated files under Docs/. README reduced
from 691 lines to 139 with links to the detailed documentation.
@Benkapner Benkapner self-assigned this Aug 13, 2026
@Benkapner Benkapner added documentation Improvements or additions to documentation help wanted Extra attention is needed labels Aug 13, 2026
Updated all user-facing references in docs, comments, docstrings,
YAML comments, and metadata. The Python package name (abevalflow)
and OpenShift namespace (ab-eval-flow) are unchanged as those are
deployed infrastructure names.
@Benkapner

Benkapner commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator Author

Added a second commit that renames all remaining "ABEvalFlow" references to "Agentic Eval Flow" across the codebase (47 files, 101 replacements).

Updated: Docs/, Python docstrings and comments, YAML comments, metadata files, test assertions, plan.md.

Kept as-is: abevalflow (Python package name, ~401 imports) and ab-eval-flow (OpenShift namespace and K8s service DNS names). Renaming those would be a breaking change requiring redeployment. @GuyZivRH

@Aliciapet11

Copy link
Copy Markdown
Collaborator

PR Review

Overall: Looks good, with a couple of minor issues to flag.

What's Good

  • README restructuring is well-executed. Went from 691 lines to ~139 by moving Gates Architecture, Submission Formats, Extensibility, and Persistence into dedicated Docs/ files. The README is now a concise overview with links to details -- exactly the right pattern.
  • harness-eval integration is properly documented. Added references in the right places: "How It Works" stage 2, Related Repositories table, and gates-architecture.md with both security scanners (Cisco + harness-eval) and both quality sources (LLM review + harness-eval).
  • The ABEvalFlow → "Agentic Eval Flow" rename is thorough (47 files, 101 replacements) and correctly preserves abevalflow (Python package) and ab-eval-flow (K8s namespace) to avoid breaking changes.

Issues

  1. Broken path in plan.md (line 176) -- The rename produces cd /Users/gziv/Dev/Agentic Eval Flow (with a space), which will break the cd command without quoting. The old ABEvalFlow had no space. Low-priority since it's a dev reference, but worth fixing.

  2. Broken path in Docs/investigations/aeh_downloadverifiererror.md -- Same space issue: the investigation doc now says the pipeline repo is at /Users/gziv/Dev/Agentic Eval Flow -- needs quoting in shell contexts.

  3. Docs/implementation_plan.md repo structure tree (line 50) -- Shows Agentic Eval Flow/ as the root directory name, but the actual repo directory is agentic_eval_flow. Cosmetic but could confuse someone reading the plan.

  4. Em-dash → double-dash style change in README -- The bullet points changed from to -- (e.g., **Skills** -- Measures...). Minor style inconsistency with the rest of Docs/ which still uses em-dashes. Not blocking, but worth being consistent one way or the other.

Verdict

The PR is solid -- the restructuring makes the docs much more navigable and the harness-eval references are placed appropriately. I'd suggest fixing the space-in-path issues in plan.md and aeh_downloadverifiererror.md before merging, but they're not blockers since those are internal dev docs.

- Fixed space-in-path in plan.md and aeh_downloadverifiererror.md
  (quoted paths, use actual directory name agentic_eval_flow)
- Fixed repo structure tree in implementation_plan.md to show
  agentic_eval_flow/ instead of Agentic Eval Flow/
- Replaced em dashes with -- across all Docs/ files for consistency
  with README style
@Benkapner

Benkapner commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed all 4 review items:

  1. Broken path in plan.md -- quoted the path and used the actual directory name agentic_eval_flow
  2. Broken path in aeh_downloadverifiererror.md -- same fix
  3. Repo structure tree in implementation_plan.md -- changed Agentic Eval Flow/ to agentic_eval_flow/
  4. Em dashes -- replaced all em dashes with -- across 17 Docs/ files for consistency with README

@Aliciapet11 @GuyZivRH

@Aliciapet11
Aliciapet11 merged commit d3ccd9c into main Aug 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation help wanted Extra attention is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants