qa: structured run names + INDEX.jsonl + backfill + find_run helper

[100yenadmin]

Closes #572.

Summary

• New: qa/scripts/{indexer,backfill_index,find_run}.py (stdlib-only), qa/INDEX_SCHEMA.md, qa/README.md
• Modified: qa/ui_playtest.sh, qa/ui_playtest_app.sh (canonical name fallback + best-effort indexer auto-append), AGENTS.md, qa/QA_TOOLS.md (route agents at find_run.py), .gitignore (/qa/INDEX.jsonl{,.new,.lock})
• INDEX.jsonl is per-developer gitignored — matches the qa/ui_playtest_runs/ ignore convention.
• Complements (does not replace) qa/scores.db: that's the curated quality verdict (~69 rows); INDEX is the raw artifact catalog (~800 rows). Cross-link via scored_in_ledger.

Canonical name format (going-forward)

<YYYYMMDDTHHMMSSZ>-<sha7>-<world>-<persona>-<provider>-<scenario>

Legacy names (nb1, gate-<sha>-<persona>, handoff-<TS>-<sha>-<scenario>) parse best-effort by regex; opaque ones (sweep-newbie) fall through to mtime + null fields.

Local verification (against /Users/lume/ClawDnD-val)

$ python3 qa/scripts/backfill_index.py --root /Users/lume/ClawDnD-val
Backfill complete: 494 entries → qa/INDEX.jsonl
  run               87
  play-state       131
  transcript      276
  elapsed:        0.71s
  opaque (no metadata, no sha in name): 2
    run/sweep-newbie
    run/sweep2-optimizer

$ python3 qa/scripts/find_run.py --since 2026-06-01 --kind run --count
41

$ python3 qa/scripts/find_run.py --scored --count
16   # cross-linked to qa/scores.db

$ python3 qa/scripts/find_run.py --gave-up --kind run --count
54

Spot-check 5 random run entries against source run.json: 5/5 cross-validated (commit_sha matches build_sha prefix).

Idempotency: re-running backfill or --append on an already-indexed dir UPDATES the row, no dupes (key = (kind, id)).

Runner-hook safety: indexer crash → exit 2; || true wrapper catches → runner continues with its own $SCORE_RC. bash -n clean on both modified runners.

Test plan

• bash -n qa/ui_playtest.sh qa/ui_playtest_app.sh — no syntax errors
• backfill_index.py against full local corpus — 494 entries in <1s, 5/5 spot-checks pass
• find_run.py exercise: --since, --sha, --persona, --gave-up, --failed, --scored, --has-rubric, --paths-only, --count, --jsonl
• Idempotency: backfill twice → identical content (modulo indexed_at)
• Indexer --append idempotency: same dir → row UPDATED, count unchanged
• Runner-hook simulation: indexer crash with bad path → exit 2; wrapped in || true → runner unaffected
• sqlite3 connection uses ?mode=ro — safe to read while scores_db.py --add runs concurrently
• fcntl flock for concurrent runners — works on macOS
• Schema in INDEX_SCHEMA.md matches what indexer.py writes matches what find_run.py reads
• Live runner smoke (one fresh ui_playtest.sh invocation, confirm INDEX appended) — owner can run in their next playtest cycle

Out of scope (follow-ups)

• Pruning / retention policy
• Renaming existing opaque dirs (sweep-newbie, etc.) — index in-place, preserves mtimes
• Engine-side auto-append for play-state/ (engine is Python, separate concern; backfill catches existing)
• SQLite migration of INDEX (JSONL fits ~800 rows + append-only failsafe + grep/jq-friendly)
• Committed shared catalog (per-dev local indexing matches the artifact-dir gitignore convention)

Confidence

~95%. The 5% empirical gap: whether the 2 opaque legacy dirs without metadata or a parseable sha (sweep-newbie, sweep2-optimizer) parse cleanly with the mtime-only fallback — backfill against the full local corpus confirms they do.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 54b8fab0-3c6a-4435-9db7-bccdd3b34f83

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}