Skip to content

feat: add exact line shell expectations - #43

Open
759132989-crypto wants to merge 4 commits into
nano-step:mainfrom
759132989-crypto:main
Open

feat: add exact line shell expectations#43
759132989-crypto wants to merge 4 commits into
nano-step:mainfrom
759132989-crypto:main

Conversation

@759132989-crypto

Copy link
Copy Markdown

What does this PR do?

Adds expect_exact_lines to kind: shell checks. It compares ordered stdout lines while ignoring trailing whitespace on each line and blank trailing lines. README and CHANGELOG are updated.

Why?

Existing shell expectations could match a flat exact string, regex, or minimum number, but could not express a readable ordered list of exact output lines.

How is it tested?

Added scripts/eval/tests/shell_exact_lines.sh with positive, reordered-line negative, and empty-output cases.

Targeted validation on macOS:

  • bash scripts/eval/tests/shell_exact_lines.sh — PASS
  • bash scripts/eval/tests/shell_no_expectation.sh — PASS
  • bash scripts/eval/tests/shell_safety.sh — PASS
  • bash -n scripts/eval/lib/score.sh scripts/eval/tests/shell_exact_lines.sh — PASS
  • git diff --check — PASS

Before / after evidence

Before: a YAML list under expect_exact_lines was not recognized as an expectation.

After: the new regression suite prints:

PASS: shell expect_exact_lines compares ordered lines and ignores trailing whitespace/blank lines


Checklist

  • I ran for t in scripts/eval/tests/*.sh; do bash "$t"; done and all suites passed
  • I added a test (or updated one) covering the change
  • I updated CHANGELOG.md under ## Unreleased
  • I read CONTRIBUTING.md
  • (If touching score.sh or attribute.sh) I checked the change works under BSD grep on macOS

The full suite was attempted under Bash 5.2 on macOS. Unrelated environment-dependent suites failed because their opencode stub/path resolution returned exit 127, and one LangGraph test used GNU-only sed syntax. I am not claiming a fully green suite.

Prepared with AI assistance in a human-supervised workspace; the human account owner remains accountable for this submission and review responses.

Closes #31.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an expect_exact_lines variant to kind: shell

1 participant