Skip to content

Qualify RSI trials with explicit completeness and lifecycle evidence #340

Open
Open
Qualify RSI trials with explicit completeness and lifecycle evidence#340
Labels
area:knowledgeKnowledge line: packs, skill library, knowledge admissionarea:workbenchCLI workbench: profile, A/B loop, quality gates, evidence reportsbot-reviewedkind:taskBounded delivery task with a delegated slice

Description

@amy-why-3459
amy-why-3459
opened on Nov 4, 2026

Current state: Contributor @amy-why-3459 has submitted ready-for-review #341, head 1cdbb70d11b4d335ba907113791c4a096beb75bd, for the bounded opt-in external-command trial checks and reusable workflow guidance described below. Selected literal-boolean adapter attestations are frozen in RunSpec and checked on both arms; these do not independently prove worker/device health, model quality or speedup. The PR was marked Ready at the preceding 8d01acd head; its description now also says ready for review. The earlier draft-stage Ready run passed its five applicable jobs and skipped KV qualification. All six jobs in the preceding 8d01acd0 Ready run passed. All six 89700595 Ready jobs, including KV qualification and both full suites, passed. All six jobs in the 1cdbb70 current-head Ready run have passed by 09:35 UTC; this software result is distinct from review clearance and author admission. The preceding 8d01acd0 declaration check failed with 'Material AI author 1: model is unapproved or suspended'. The earlier declaration check failed on stale/invalid metadata. The PR now binds exact head 1cdbb70, preserves both Codex and Cursor/Grok commit-attributed contributions with unknown exact model/effort, and requests an exception across all protected changed paths. The new declaration check fails because independent owner exception approval is absent and both material authors' model identities are unapproved; stale head is no longer the current failure reason. Current-session settings are not substituted for unknown historical provenance. Passing software CI does not resolve admission. The earlier scoped COMMENT review now reports one P2 evidence-presentation finding: failed/missing required trial checks suppress improvement_pct and yield INCONCLUSIVE, but the HTML/SVG consumer can still plot an apparent gain from raw medians. The reviewer requests suppression or unmistakable raw/unqualified labeling plus a report-level regression. The author's correction at 89700595 now reports a persisted trials_qualified flag, no derived gain chart for unqualified comparisons, raw-measurement labeling and report-level regressions. The six-file delta changes evaluation, reporting, chart handling, tests and documentation. The original reviewer's follow-up found a remaining pre-existing summary-chart.json path that bypassed the first repair. The author's second correction at 1cdbb70 now reports guarding both chart paths, retaining historical spec bytes without drawing them, and a regression for that exact case. The original reviewer's scoped recheck now clears this presentation finding at exact head 1cdbb70: the guard precedes both chart paths, historical JSON remains unchanged and raw medians stay labeled. It records all six Ready jobs passing and both full suites at 3,571 passed / 13 skipped, tested merge 38cf7fb with a tree matching the head. The same recheck confirms current-head author admission remains noncompliant under main 9dded7a; this is not whole-PR approval or acceptance. This synchronization reused that review without rerunning tests or duplicating it. The PR's earlier local review/evidence and disclosed failures remain separate. This is submitted implementation, not maintainer integration, goal acceptance or hardware/model qualification.
Decision needed: An authorized independent policy-approval owner must disposition the exact-head exception request; declared unknown historical identities remain unknown. Maintainer review and goal acceptance of the submitted delivery remain outstanding.
Next action: Existing PR author @amy-why-3459 retains exact-head policy-exception and remaining whole-PR review/acceptance follow-up; the specific report-presentation repair has received its scoped recheck. Track current-head policy/review disposition without duplicating implementation; preserve the stated adapter-attestation and hardware-evidence limits. This status update does not authorize new scope, execution, spending, merge or closure.

Goal and accepted scope

The contributor requested an InferRSI PR that incorporates workflow lessons from Omni model integration: reference parity, effective graph controls, streaming boundaries, and complete paired experiments with clean lifecycle evidence. Implementation and PR submission are delegated to Codex; merge and Issue closure remain with the owner/maintainers.

Bounded delivery

  • Add opt-in trial qualification checks to the external-command harness, selected through CLI and frozen in RunSpec. Each baseline/candidate trial must provide literal-boolean evidence for the selected checks; missing, malformed, or false checks make the comparison INCONCLUSIVE and cannot support a speedup claim.
  • Preserve the existing execution status, quality validation, source integrity, AB/BA order, and unconfigured command compatibility. Checks are adapter attestations backed by retained receipts, not automatic proof that devices or detached workers are healthy.
  • Document reference numerical/RNG contracts, feedback-aware stage boundaries, matching full/stream reference paths, actual internal/outer graph switches, metadata-only/final-empty payload preflight, shared-device qualification, and full-comparison reruns after harness repair.
  • Add CPU regression tests and update the discoverable RSI evidence workflow. Do not add campaign constants, default accelerator jobs, model dependencies, or a new CI lane.

Acceptance

  • Missing/false/non-boolean checks on either arm cannot produce PASS or a qualified improvement.
  • Complete checked trials can pass existing gates; commands without selected checks remain compatible.
  • CLI -> RunSpec -> persisted trial evidence -> evaluation is exercised with real synthetic commands.
  • Documentation distinguishes framework tests, adapter claims, and actual hardware/model qualification.

Related source: vllm-project/vllm-omni#8972 (workflow updates). This task makes no model-quality or hardware-performance claim.

Activity

  1. agent-ci-infra-bot commented on Nov 4, 2026

    @agent-ci-infra-bot
    agent-ci-infra-botBot
    on Nov 4, 2026 – with agent-CI-Infra-bot

    Intent: add opt-in trial-qualification checks to the external-command harness — CLI-selected, frozen in RunSpec, persisted per baseline/candidate trial — so missing, malformed or false attestations make the comparison INCONCLUSIVE instead of supporting a speedup claim, with CPU tests and RSI evidence documentation. The named code surfaces exist, but the check set, CLI/evidence schema and the repository-required status/decision record are missing. Verdict: needs information / human decision required; do not start development yet.
    Labels: kind:task, area:workbench, area:knowledge (bounded harness delivery plus the RSI evidence workflow; no priority because the Issue has no status:proposed/status:accepted label or recorded owner decision).

    F1 — needs information — the selected check set and per-trial evidence contract are undefined

    • Where: src/inferrsi/runspec.py:9-35 (no check field), src/inferrsi/arguments.py:47-104 (no CLI selector), src/inferrsi/harness/scenarios/external.py:12-32 (writes only execution.json and the metric), src/inferrsi/harness/evaluation.py:44-66 (gates only on trial status, metric, validation and integrity).
    • Problem: The Issue requires "selected checks" and "literal-boolean evidence" backed by "retained receipts", but it never lists check IDs or meanings, the CLI spelling, the persisted file/key names, or which JSON values count (true vs "true" vs 1). The external harness has no receipt contract; the only receipt mechanism in the tree is Skill Eval's receipt.json (docs/skill-eval-comparison.md:217), a different lane. The linked vllm-omni PR is documentation for another repository, not a schema authority.
    • Conditions: Any implementation of the bounded delivery.
    • Impact: The implementer must invent a public contract that is frozen into run.lock.json (src/inferrsi/harness/runner.py:34) and trial evidence; regression tests cannot assert the intended behavior, and reviewers cannot check the acceptance bullets.
    • Fix: Add a short interface spec: the flag name and repeatable/comma-separated check IDs; the exact per-trial attestation filename and JSON keys; the accepted boolean representation; evaluation reason strings; and one positive plus one failing synthetic example. Split out a design step first if the semantics are still open.

    F2 — needs information / human decision — the required Issue status block, labels and a recorded acceptance are missing

    • Where: .github/ISSUE_TEMPLATE/scoped-task.md:17-19, docs/agent-collaboration.md:191-194,212-216.
    • Problem: The body has no **Current state:**, **Decision needed:**, **Next action:** block, no kind:* or area:* labels, and the Issue has zero comments. It claims "accepted scope" and delegates implementation to Codex, but status:accepted requires the decision owner to record that decision in a comment (docs/agent-collaboration.md:194), and delegation must be recorded or linked (docs/agent-collaboration.md:15).
    • Impact: The owner and the authorization are unverifiable, and the repository's task-issue convention is unmet, so unattended work should not start.
    • Fix: Add the three status lines, name the decision owner and the acceptance/delegation link, and apply kind/area labels (suggested: kind:task, area:workbench, area:knowledge).

    F3 — needs information / human decision — one Issue mixes harness code with a broad docs/skill delivery, including protected knowledge

    • Where: docs/knowledge-model-policy.md:31-43, src/inferrsi/data/skill_library/workflows/rsi-evidence/SKILL.md:1, docs/agent-collaboration.md:97.
    • Problem: "update the discoverable RSI evidence workflow" most plausibly means the rsi-evidence skill, which lives under src/inferrsi/data/skill_library/** — protected scope requiring an approved model at xhigh/max and a knowledge-model-policy PR declaration. The Issue delegates to Codex without naming a model or effort, and it does not distinguish the skill from ordinary docs such as docs/contributing.md or docs/quickstart.md. The docs bullets (reference numerical/RNG contracts, stage boundaries, graph switches, preflight, shared-device qualification, reruns) are a separate delivery from the harness check.
    • Impact: A combined PR can fail the required knowledge-model-policy status, and the mixed scope is hard to review against the acceptance bullets.
    • Fix: Split the code and the docs/skill delivery into separate PRs, or name the exact files and state the approved-model/declaration plan; make the rsi-evidence skill explicitly in or out of scope.

    F4 — scope risk — compatibility acceptance does not cover persisted RunSpec readers and shared-runner adapters

    • Where: docs/architecture.md:4 (RunSpec and command contracts "remain stable"), src/inferrsi/harness/runner.py:34, tools/qualify_kv_reclamation.py:165-181, examples/qwen21_turbo_validation_adapter.py:21.
    • Problem: A new RunSpec field and evaluation gate become part of the persisted run.lock.json and of the shared runner used by the synthetic KV qualifier and legacy validation adapters. Acceptance only says "commands without selected checks remain compatible"; it does not require absent-field/default-off behavior or consumer regression coverage.
    • Impact: If checks default to required, or an absent check record counts as incomplete, these existing tool/adapter workflows become INCONCLUSIVE.
    • Fix: State that an absent check selection is default-off and add a regression test that exercises tools/qualify_kv_reclamation (or an equivalent unconfigured adapter) through the new evaluation.

    F5 — related work — PR #202 already tightened the same harness verdict/state path, and Issue #55 may overlap

    Reviewed commit: 663f3b3
    Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

  2. agent-ci-infra-bot
    added
    area:knowledgeKnowledge line: packs, skill library, knowledge admission
    area:workbenchCLI workbench: profile, A/B loop, quality gates, evidence reports
    bot-reviewed
    kind:taskBounded delivery task with a delegated slice
    on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:knowledgeKnowledge line: packs, skill library, knowledge admissionarea:workbenchCLI workbench: profile, A/B loop, quality gates, evidence reportsbot-reviewedkind:taskBounded delivery task with a delegated slice

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions