Skip to content

Pin and document end-to-end: 'run a real eval-harness against omo-session-distiller' #25

Description

@hoainho

What

KNOWN_ISSUES.md notes 'No baselines ship in-repo.' First-time users running the README's 5-min quick start get verdict: FAIL (no baseline diff). This is correct behavior but a bad first impression.

Design

Two parts:

Part 1: A scripted demo

Add scripts/demo.sh that does the whole tutorial end-to-end:

#!/usr/bin/env bash
# Walks a new user through baseline → edit → detect-regression
set -e
echo "== Step 1: baseline omo-session-distiller (uses your real ANTHROPIC_API_KEY) =="
eval-harness baseline --skill=omo-session-distiller
echo
echo "== Step 2: mutate the skill (drop a tag) =="
# Edit a known line
sed -i.bak 's/architecture,database/architecture/' skills/omo-session-distiller/SKILL.md  # example
echo
echo "== Step 3: re-run; harness should detect regression =="
eval-harness run --skill=omo-session-distiller
echo
echo "== Step 4: restore =="
mv skills/omo-session-distiller/SKILL.md.bak skills/omo-session-distiller/SKILL.md

Part 2: a recorded asciinema cast under docs/demo.cast (or animated gif) that a contributor can replay without their own API key.

Scope

demo.sh itself is small (~30 lines). The asciinema cast is nice-to-have but slows down review. Start with just the script + a brief README pointer.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationgood first issueGood for newcomersv0.5.0CI-ready release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions