[Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments

@Bounty-hunter

Current state: @Bounty-hunter remains the assignee and #338 remains Draft, now at 6ed93bb41f7db9dafb420dcb6f8705f1864bc45c. The updated contributor index records a two-line work split: #336 tracks continuing task-library batches; #337 tracks real-Agent/external-task integration and whole-run result/cost association. #334 and #335 are CLOSED / NOT_PLANNED for consolidation, not delivered-function or project acceptance. Their history remains preserved. The author plans existing #338 plus one later draft integration PR, which has not been created; this status records that plan without launching or accepting new work. The author index reports user-directed NPU priority and a completed trusted-source Ascend 910B1 RMSNorm software-control pilot; maintainer scope and project acceptance remain unrecorded. Published evidence retains tested source 6308268fc6d1fbf90e9ab97b5da8af79152f39ea: baseline/reference each pass 375 assertions, six wrong controls and an extra post-precheck replay are rejected, and two Workbench comparisons contain 16 trials / 800 timing samples. Recomputing the published samples reproduces per-shape A/A differences of 0.374% / 0.296% and fused-reference reductions of 34.802% / 30.603%; these are synchronized eager completion-call latencies for float32 [1,4096] / [64,4096], including dispatch and synchronization, not pure-kernel or model/service speedups. A/A's raw FAIL means no qualifying gain; final correctness passes. The reference comparison's raw PASS supports the author's bounded keep decision for an existing fused implementation, not new-kernel discovery or Agent/skill efficacy. Device execution, trace provenance and scheduler release remain author-reported; this synchronization checked public sample arithmetic, not private raw traces or a hardware rerun. The a91 delta changes documentation/evidence only.

The original reviewer has cleared the specific canonical-task shadowing finding at 6308268f, citing canonical descriptor/starter/verifier/prepare loading, cwd-shadow regressions and 54 real Linux artifact controls with zero skips. This scoped clearance remains distinct from whole-PR approval. The separate P2 NPU action/total-timeout finding remains unresolved: paired trials do not receive the frozen 180-second action cap, and the script does not itself enforce the stated 20-minute pilot envelope. The public evidence records 8-minute comparison and 180-second action settings plus scheduler completion, but supplies no enforceable outer-deadline receipt resolving that finding. All six applicable current-head Ready checks and declarations passed; draft KV qualification skipped. Hosted CI is software evidence, not NPU execution. The preceding 12-file docs-only delta makes profiling optional in the general flow while preserving the executed pilot's frozen profiling checks, tested source and 800 samples; it changes no timeout code. The preceding 19-file delta removes the built-in rmsnorm-cpu package and its dedicated probe, migrates shared regression controls to softmax-cpu, and retains rmsnorm-npu; the NPU implementation, shared grader and original device-evidence identity are unchanged. Historical CPU records remain tied to their original checkout, not silently reassigned to the retained tasks. The 6ed93bb4 cleanup removes docs/plans/issue-252 from the current tree, consolidates task operations in docs/engineering-benchmarks.md, and adds docs/engineering-task-artifacts.md for CPU handoff. Plans/progress now live in the Issues; all prior research and experiment records remain pinned at 9a755f97. The current PR is reduced to 39 files, +3901/-75 lines. The change list leaves runtime modules, task packages and tests unchanged; the only Python utility edit is its module docstring. Author-reported archive-byte/AST checks are not independent hardware replay by this synchronization, and the cleanup does not fix the timeout or establish durable upstream archival acceptance. The 9a755f97 review found the CPU removal/test migration consistent. The later 6ed93bb4 cleanup review now explicitly treats the evidence-size/byte-budget question as answered by removing the dumps, while retaining two concerns: enforcement of paired-action/total NPU timeouts, and a named durable location for the historical CPU/NPU evidence after merge. The deleted walkthrough links are no longer current source references; the historical records remain accessible at the cited fork revisions, which is distinct from an agreed long-term retention location. The reviewer has not approved this head. The retention-location request does not itself alter accepted storage or execution requirements. The review's suggestion that Ready checks await removal of Draft is superseded by the actual successful run linked above; KV qualification alone was skipped. This synchronization reused the static review and hosted checks without rerunning software or hardware. Real-Agent NPU execution, external-project task consumption and complete run-cost linkage remain undelivered. The author reports label-write permission denial during consolidation; old closed-item labels and #336's kind label may therefore lag the recorded Issue states, and this synchronization does not alter labels.

Earlier CPU v1/v2 measurements, failures and formal correctness-artifact controls retain their original identities in the historical delivery record. The formal CPU engineering-benchmark-v1 adapter can accept an unchanged correct starter and explicitly rejects NPU; it does not establish timing discrimination, producer permissions or NPU artifact isolation. Formal NPU isolation, controlled Agent/skill benefit, provider generation, skill_eval_improve, broader task/dtype/shape coverage and model/service qualification remain open. The automatic SkillEval plan selected zero comparisons and was not an efficacy run. Contributor-reported delegation does not expand this synchronization's authority or replace maintainer acceptance; the original tracker and checklist below remain unchanged.
Decision needed: Requester/acceptance owner @Dong1017 to record acceptance or revision of the task-library batch and bounded end-to-end integration deliveries now tracked by #336/#337. The recorded consolidation and closure reasons do not establish project acceptance or merge approval.
Next action: Existing assignee @Bounty-hunter retains disposition of the NPU timeout review and current-head re-review, alongside the recorded scope decision and remaining #336/#337 qualification; #248 and #249 keep their existing ownership. Preserve the cleared shadowing finding separately from the unresolved timeout finding and preserve CPU/NPU evidence identities. The separate optimization-search-policy RFC remains separate under the earlier design review. This status update authorizes no new implementation, hardware/model execution, spending, merge or closure.

Goal

Track a minimal, reusable engineering benchmark foundation for InferRSI, organized around SearchSpace and engineering objectives, not around individual skills.

Reuse and adapt existing benchmarks wherever possible. The shared task/environment/validation packages should support:

Workbench measurement of code/configuration candidates;
SkillEval controlled agent comparisons for skill/knowledge PRs;
overall harness experiments (task: measure whether InferRSI helps Claude Code and Codex optimize inference tasks #248) and optimization-search experiments (task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249).

A benchmark measures the produced implementation. Attributing an improvement to a skill or search policy additionally requires a controlled experiment; these are separate responsibilities.

Candidate coverage, to narrow before implementation

Operators: representative shapes/dtypes, reference correctness and numerical tolerances, latency/throughput.
Communication: topology and message-size conditions, correctness, latency/bandwidth.
Memory: representative workloads, peak allocation/reservation definitions, capacity/OOM boundaries, with correctness and performance constraints.
Further SearchSpace slices only when justified by an actual campaign or accepted requirement.

These are coverage directions, not a commitment to build every category or hardware combination in the first delivery. No per-skill large question bank or universal aggregate score is required.

Minimal shared contract to discuss

For each selected task: pinned upstream source/license, workload and allowed search scope, environment/device requirements, baseline/reference implementation, independent validation commands, metric units and timing boundaries, repeatability checks, and explicit unsupported/missing coverage.

Keep task/grader versions fixed within a comparison. Reuse existing runners and evidence mechanisms rather than introduce another execution platform. CPU/software checks do not establish GPU/NPU correctness or performance; unavailable environments remain unverified.

Relationship to existing work

[RFC] Benchmark selection by task for InferRSI #60 (closed): task-specific benchmark selection research, especially model/serving workloads. Reuse its source report; this tracker concerns the shared minimal engineering benchmark delivery.
[RFC] Iteration Campaign Plan: one table for model, hardware, goal, search plan and test plan #30 and [RFC] Hardware-aware optimization goals throughout the RSI workflow #143: campaign plans, hardware-aware goals and constraints.
[Tracking] SearchSpace distillation coverage and knowledge consolidation #91: SearchSpace knowledge coverage, not benchmark execution coverage.
Skill Eval v1.1: automated evaluation for skill additions and updates #88: SkillEval comparison/reporting workflow; reference supplement. Benchmark construction remains separable from SkillEval implementation.
task: measure whether InferRSI helps Claude Code and Codex optimize inference tasks #248: overall agent benefit experiment; task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249: parallel candidate-search scheduling. Neither alone supplies this shared benchmark foundation.

Initial reuse candidates include KernelBench, BackendBench, existing framework tests/benchmarks, and applicable task-package practices from SkillsBench. Listing a source is not adoption or evidence that it supports the target device.

Tracking and acceptance

Agree a small initial SearchSpace/task subset, upstream reuse choices and accountable implementer.
Agree the concrete task/environment/oracle interface and scoped delivery criteria before implementation.
Deliver and validate the selected subset, retaining reproducible measurements and explicit coverage limits.
Demonstrate that the selected shared package can be consumed by its agreed Workbench/evaluation use cases without duplicating task truth or runners.
Record evidence-backed acceptance of the agreed subset; do not close merely because a catalog/document exists or software CI is green.

Ownership and authorization

Requested by @Dong1017 as an umbrella tracking issue. Current authorization is to track and clarify this work, not to launch implementation, claim another contributor's work, run paid models or allocate hardware. Implementer and initial subset remain unassigned. This issue does not reopen completed scopes or alter #248/#249 ownership.

Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation.

[RFC] Optimization search policy: candidate admission, cost ladder and stop contract

Status: Proposed. Base revision a8215b3.
Creates no threshold, revision pin, campaign, promotion or execution authority.
Related: this comment on #252; depends on #252 for task/oracle packages, orthogonal to #249 (parallel lanes), extends row 4 of the campaign plan RFC #30.

Item	Value
Scope	Machine-readable search-space projection; candidate card schema; admission gates; cost tier → `RunSpec` mapping; phase-locked round recording; stop evaluator; negative-result ledger
Out of scope	Candidate synthesis, code editing by a model, autonomous execution, agent-side hook enforcement, `evaluation.py` statistics, parallel lanes (#249), benchmark task packages (#252)
Reuses	Campaign plan row 4 field set; `RunSpec` (`src/inferrsi/runspec.py`); AB/BA runner; `assessment()` short-circuit shape (`src/inferrsi/harness/improvement.py`); existing rejected-direction cases
Adds	No 🟢 SearchSpace cell, no measured coverage, no skill, no claim

1. Problem

optimization-search-policy already states a complete policy: four admission considerations, a seven-phase locked round, a six-tier cost ladder and four stop criteria. Every one of those is prose. Nothing in src/inferrsi/ reads it.

The gap is visible in three concrete places:

Row 4 of the campaign plan already specifies the candidate card — ID, plane, target stage and its measured exposed share of E2E, conditional headroom, diagnostic, hypothesis, mechanism class, risk class, cost tier, prior art or negative record, kill criterion, status, evidence link. It is a human-filled table cell. There is no schema, no validator and no way to ask "which cells are still admissible".
Prior-art and negative-record lookup is unimplemented and, as written, contradicted. The policy requires recall-complete free-text grep over full case bodies, while src/inferrsi/catalog.py:127 scores records by token-AND (every token must appear) and ranks by raw occurrence count. That is a precision-ordered substring index, not a recall-complete case search. The three existing rejected-direction cases are plain Markdown with no query interface.
The stop criteria have no inputs. S1 needs measured uncertainty and removable time; today evaluation.py:36 reports range_pct_of_median as descriptive only and asserts "statistical_significance": "not established". S3 needs a verdict history across rounds; nothing aggregates rounds.

The result: the policy is correct and unusable. An agent can read it and still re-enter a recorded dead end, because nothing makes the dead end queryable at the moment of candidate generation.

The narrow claim of this RFC: the admission, ordering and stopping decisions can be made deterministic over declared inputs and existing evidence, without synthesizing candidates and without spending compute. That is a contract problem, not a modelling problem.

2. Design principles

Row 4 is the candidate card. Do not introduce a second ontology. This RFC defines a machine-readable form of row 4 plus the evaluation over it, and defers to docs/searchspace-matrix.md as the authoritative coverage matrix. No third matrix.
Gates produce refusals, not rankings. A gate can kill a candidate, mark it unadmissible without a diagnostic, or raise its burden of justification. It never scores two candidates against each other. Ranking by expected value is a further decision that needs the measurement contract to exist first.
Absent input stays absent. Every gate that lacks an input returns unverified with the cheapest check that would resolve it. No default fraction, no assumed share, no fallback threshold. This matches the existing INCONCLUSIVE > FAIL > PASS precedence.
Immutability follows row 4's freeze semantics. A kill criterion and a cost tier are fixed once the candidate has run; changing either requires a new candidate ID, not an edit.
The policy governs records, not agent behaviour. vllm-evolve enforces phase order with a PreToolUse hook. InferRSI cannot and should not: it is not in the agent's tool path. The enforceable analog is that a round which skips a phase cannot be recorded as a qualifying round, and an unrecorded round cannot feed a promotion or a stop decision.

3. Objects

3.1 Search-space cell index

Generated from docs/searchspace-matrix.md by extending tools/build_skill_coverage.py (which already resolves SEARCHSPACE = Path("docs/searchspace-matrix.md")), never hand-authored. The matrix remains authoritative; the index is a derived artifact with a --check mode that fails on drift.

{
  "schema_version": 1,
  "id": "accelerator.kv.admission",
  "layer": "accelerator",
  "stages": ["diagnose", "configure", "verify"],
  "target_classes": ["serving", "kv-cache"],
  "skill_ids": ["diagnose-vllm-admission-pressure"],
  "evidence_pins": [{"source": "<repo>", "commit": "<40-hex>"}],
  "maturity": "source_candidate",
  "plane_hint": "model",
  "pruning_claims": [
    "Queue length and KV usage do not establish allocation pressure",
    "stats are gated and incomplete"
  ],
  "rejected_direction_refs": ["casebook/cases/qwen-ascend-rejected-directions.md"]
}

pruning_claims is the load-bearing field. The matrix's Cases / Experience column currently holds falsifying statements that only a human can read — "Shape agreement is not conversion", "CPU bypasses accelerator capacity and segmentation gates", "matching total tokens does not preserve peer rows". Those are pre-registered counter-claims. Surfacing them as queryable constraints is the cheapest available reduction of the candidate space, and it costs no measurement.

maturity carries the existing key unchanged (covered / partial / gap); source_candidate and not_run remain distinct states and neither becomes an admissibility signal.

3.2 Candidate card

The machine form of row 4. Field names are the campaign plan's; this adds typing, not vocabulary.

{
  "schema_version": 1,
  "id": "D-01",
  "plane": "model",
  "cell": "accelerator.kv.admission",
  "hypothesis": "If admission accounting charges <mechanism>, <metric> improves because <causal step>",
  "mechanism_class": "ownership",
  "risk_class": "lossless-exact",
  "cost_tier": "profile-run",
  "stage": {"name": "<stage>", "exposed_share_pct": null, "share_source": "unmeasured"},
  "headroom": {"removable_ms": null, "assumptions": [], "uncertainty_ms": null},
  "diagnostic": {"cost_tier": "profile-run", "kill_criterion": "<observable>"},
  "prior_art": [],
  "negative_record": [],
  "falsification": {
    "metric_key": "<dotted.positive.key>",
    "direction": "minimize",
    "criterion": "<the observation that would reject this hypothesis>"
  },
  "kill_criterion": "<immutable once run>",
  "status": "planned"
}

Validation rules (all reject, none warn):

falsification.criterion nonempty, and falsification.metric_key resolves to a positive finite number under the existing metric contract (model/protocols/metrics.py). A candidate that no single metric could reject is not a hypothesis; it is a wish.
kill_criterion nonempty and not equal to the falsification criterion.
plane ∈ harness | model | infra, one per candidate (AGENTS.md RSI contract).
cell must resolve in the generated index; mechanism_class must be one of the seven declared tags.
stage.exposed_share_pct may be null only when stage.share_source == "unmeasured" and a diagnostic is attached.

3.3 Admission gates

Ordered cheapest-first. Each returns admit | refuse | admit_conditional | unverified, with a reason and, where applicable, the cheapest resolving action.

G0 — Amdahl plausibility.

upper_bound = exposed_share(cell) × plausible_removal(mechanism_class)
refuse if upper_bound < residual_target

residual_target is campaign-declared, never a tree default. This is an upper bound on what the stage could yield, not a prediction, and it is the cheapest way to discard a candidate before any compute is spent. Missing share → unverified with required_action: "run the cheapest diagnostic in this cell" (typically a profile run, which is a measurement, not a comparison).

G1 — Mechanism class alignment. The declared mechanism_class must be reachable from the measured bottleneck class. A residency candidate against a compute-bound stage is refused with reason: class_mismatch. The five/six families the policy names — ownership, work elimination, residency, algorithm, configuration, fusion/operator, graph/launch, parallelism/communication, precision/approximation — are the declared tag set from row 4 and are not extended here.

G2 — Prior art and negative record. Two lookups, deliberately different:

Prior art (recall-oriented). Full-text grep across case bodies, per the policy's explicit requirement. Result is a list of leads.
Negative record (classification-oriented). Structured query over the ledger (§3.7), keyed by (cell, mechanism_class, backend, device, model, workload, evidence_level).

On a negative-record hit the gate returns admit_conditional, never refuse. The card must then name:

the rejection class — one of quality_rejection, no_demonstrated_gain, resource_failure, confounded_control, unintegrated_lead, analytical_rejection;
what changed since that record (revision, device, workload, evidence level);
the cheapest check that would distinguish the two conditions.

A keyword match is a lead, not a causal diagnosis and not an automatic veto. A different implementation failing does not ban a mechanism family.

G3 — Cost tier assignment. The cheapest tier that can separate the hypothesis from local noise, declared before measurement. A tier above the minimum needs a stated reason. Escalation is one-way and only on a positive result at the current tier.

3.4 Cost ladder → `RunSpec`

The policy's six tiers map onto RunSpec knobs. The mapping is shape, not numbers; tier presets are campaign-scoped in rows 4 and 6.

Tier	`RunSpec` shape	May produce a verdict?
`static/config`	no benchmark command; validation command only	correctness only — never a performance verdict
`microbench`	narrow `metric_key`, single scenario, minimal `repetitions`	screening only
`profile-run`	measurement run, no A/B pair	no verdict; supplies `exposed_share` for G0
`scenario@c1`	full paired `RunSpec`, one declared regime	screening PASS/FAIL; not acceptance
`multi-regime`	paired `RunSpec`, declared regime set	may support acceptance if rows 3/5 declare it
`full-cohort`	the frozen acceptance regime	the only tier that can feed a public claim

Two hard rules:

Escalate only on positive. A negative or inconclusive result at tier k does not buy a run at tier k+1.
Screening may reject early; it may never substitute for the acceptance regime. A scenario@c1 PASS is not an acceptance PASS, and no promotion rule may treat it as one. This mirrors the CRISP adaptation boundary already recorded in the policy: screening rejects, it does not conclude.

3.5 Phase-locked round

Seven phases, with the existing runner as P5's executor. The lock is enforced on the record:

flowchart LR
  P1["P1 Read context<br/>cell + prior art + negative record"] --> P2["P2 Design<br/>candidate card"]
  P2 --> P3["P3 Apply<br/>bounded, one plane"]
  P3 --> P4["P4 Verify correctness"]
  P4 --> P5["P5 Evaluate<br/>AB/BA at declared tier"]
  P5 --> P6["P6 Compare<br/>better / worse / inconclusive"]
  P6 --> P7["P7 Keep or discard<br/>ledger entry, unconditional"]
  P7 --> P1
  P4 -.->|"fails"| L["Ledger: correctness_rejection<br/>no benchmark budget spent"]
  L --> P1

A ledger entry is qualifying only if it records P1 inputs (the cell and the negative-record lookup result), a P2 card, a P4 outcome, and a P5/P6 result. An entry missing P4 is recorded as unqualified and cannot be cited by a stop decision or a promotion.

This is weaker than a hook by design: it cannot stop an agent from running a benchmark out of order, it can only refuse to treat the result as evidence. Given the repo's non-autonomous posture, that is the correct strength.

3.6 Stop evaluator

Deterministic over the ledger. Each criterion yields {decision: stop | continue | unverified, reason, required_action}.

ID	Criterion	Inputs	Behaviour when input missing
S1	Current-mechanism headroom exhausted	removable exposed time, `uncertainty`	`unverified` — never infer headroom from a roofline
S2	Amdahl ceiling reached	residual target, plausible removal over remaining stages	`unverified`
S3	Repeated inconclusive on the same hypothesis	count of consecutive `inconclusive` for one `hypothesis` hash	`continue` with `required_action: "repair the measurement protocol or change hypothesis"`
S4	Negative record saturated	every cell in the declared scope has a terminal ledger entry	`stop` with `required_action: "escalate abstraction level"`

S1 must not be satisfied by proximity to a roofline. The existing policy says this explicitly and it matters: changing the algorithm changes both work and traffic, so proximity to the old roof does not rule out improvement. The evaluator encodes that by requiring a measured removable quantity, and returning unverified otherwise.

S3's repetition count is campaign-declared. No tree default.

3.7 Negative-result ledger

One record per terminal round outcome, including worse and inconclusive. This is the compounding asset: the repository already keeps three rejected-direction cases by hand, and each one exists because someone paid to learn it.

{
  "schema_version": 1,
  "candidate_id": "D-01",
  "cell": "accelerator.kv.admission",
  "mechanism_class": "ownership",
  "context": {
    "repo": "<repo>", "revision": "<40-hex>", "backend": "ascend",
    "device": "<exact SKU>", "model": "<model>", "workload": "<id>"
  },
  "evidence_level": "scenario@c1",
  "hypothesis": "<as tested>",
  "metric_key": "<key>",
  "verdict": "FAIL",
  "rejection_class": "no_demonstrated_gain",
  "observed": "<measured values and uncertainty>",
  "constraint_derived": "<the search constraint this failure establishes>",
  "reopen_condition": "<what changed would justify retrying>"
}

constraint_derived is mandatory. A negative result that establishes no constraint has not narrowed the space and cannot satisfy S4.

4. Relationship to existing work

Work	Relationship
#252	Supplies the task package and oracle that P4/P5 consume. This RFC defines which candidate is admissible and what tier measures it; #252 defines what the measurement runs against. Neither duplicates the other's truth.
#249	Parallel lanes are an orthogonal scheduling change. A lane submits a candidate card; the round records it. Nothing here assumes one lane.
#30 (campaign plan RFC)	This RFC is the schema and evaluator for row 4, and defers to rows 1/2/3/5/6/7 unchanged. It proposes no new frozen row.
#248	Harness benefit experiment. The ledger feeds it with per-round outcomes; this RFC does not run it.
#91	SearchSpace knowledge coverage. The cell index is derived from that coverage, not a second copy of it.
#88	SkillEval. Its `assessment()` short-circuit ordering (`incomplete_evidence → unknown_usage → quality_floor → regression_limit → no_improvement`) is the precedent for §3.3's ordering, not a component to reuse: it scores skill-text edits, not inference candidates.
`optimization-search-policy` skill	Promoted from prose to contract. The skill remains the human-readable rationale and gains references to the generated schemas.

5. Dependency and ordering

The gates can land first; multi-round search cannot.

Two preconditions for using this contract across more than one round:

Measurement resolution. harness/evaluation.py:36 compares two independent medians and discards the pairing that the AB/BA schedule already established. Over repeated rounds, selecting the best of N unvalidated comparisons is selection bias, not tuning, and it will deposit noise into the ledger as if it were a constraint. The minimum repair — paired difference plus a resolvability gate that refuses PASS when baseline spread is on the order of min_improvement_pct — is a change to evaluation.py, not to this RFC, and is not proposed here as a slice.
Candidate-space indexing. Without the cell index and the negative-record query, G0/G1/G2 return unverified for everything and the gates reduce to a schema validator.

So the slices below are ordered accordingly, and no slice depends on statistical work.

I am not claiming that fixing resolution is sufficient. It is a precondition for round count > 1, and stating it here is cheaper than discovering it after a campaign.

6. Delivery slices

Slice	Content	Depends on	Hardware
A	Cell index generated from the matrix; `--check` drift gate	—	none
B	Candidate card schema + validator; rejects unfalsifiable cards	A	none
C	Negative-record ledger schema + query over the three existing cases	A	none
D	G0–G3 evaluation; `unverified` paths	A, B, C	none
E	Cost-tier → `RunSpec` mapping + escalation refusal	B	none
F	Round recorder: qualifying vs unqualified entries	B, E	none
G	Stop evaluator S1–S4 over a ledger fixture	C, F	none
H	One synthetic end-to-end fixture: cards in, gate decisions and stop decision out	D, G	none

Every slice is model-free and hardware-free. No slice spends compute, allocates a device, calls a paid model, or produces a measured claim.

7. Acceptance criteria for this RFC

A1. The generated index round-trips against the matrix, and --check fails on an unreconciled edit. The matrix remains the only authoritative source; the index cannot be edited by hand.
A2. The card validator refuses: no resolvable positive-finite metric; empty kill criterion; kill criterion equal to the falsification criterion; more than one plane; an unresolvable cell; a null stage share with no diagnostic attached.
A3. A negative-record hit yields admit_conditional, never refuse, and a card without what changed fails to validate.
A4. A card cannot be assigned a tier above the minimum without a stated reason, and a scenario@c1 result cannot satisfy a promotion rule.
A5. A round missing P4 is recorded unqualified and is rejected as input to S3/S4.
A6. S1 and S2 return unverified — not continue — when headroom inputs are absent.
A7. A negative entry with an empty constraint_derived cannot satisfy S4.
A8. One synthetic fixture demonstrates that the gates select the subset a human would, and that the fixture contains no measured performance value.
A9. No new 🟢 cell, no measured coverage, no skill promotion, no claim. Every new artifact is status: candidate or a schema.

8. What this RFC does not authorize

No execution, device allocation, paid model call or benchmark run.
No change to evaluation.py, the runner, the AB/BA schedule or any frozen row.
No numeric default enters the tree. Repetition counts, tier presets, residual_target and S3's N are campaign-scoped in rows 4 and 6.
No promotion of any knowledge, skill or SearchSpace cell.
No claim that the policy improves search outcomes. The policy's own benefit is unmeasured and this RFC does not establish it.

9. Open questions

Does the cell index belong in src/inferrsi/data/ as package data, or stay a tools/ artifact? Package data would let knowledge search join on cells; it would also create a second shipped copy of the matrix.
Should G2's prior-art lookup use the existing token-AND catalog or a separate grep path? The policy demands recall-complete free-text search, and catalog.py:127 is precision-ordered by construction. Separate paths risk the two disagreeing.
How is plausible_removal(mechanism_class) bounded without inventing numbers? Options: campaign-declared per class, derived only from recorded positive results in the ledger, or left unverified until a campaign declares it. The third is honest and the least useful; the second cannot bootstrap on an empty ledger.
Does an unqualified round still write a negative ledger entry? It carries information but fails the P4 precondition. Recording it pollutes S4; discarding it loses a real dead end.
Should the ledger be issue-scoped or in-tree? Campaign parameters and numeric gates stay issue-scoped per AGENTS.md; a ledger that only lives in an issue is not queryable by the gate that needs it.

Proposed for review. Comment on scope and slice boundaries before any implementation claim. No part of this proposes reopening #248, #249 or #252.

@Dong1017

Issue/design review of the body updated 2026-10-29T11:14:20Z and the optimization-search-policy RFC, against main 7d1d82f55cda89842bd23ee2da91b521ee60eb08. Source/document inspection only; no benchmark, model, device or workflow execution.

The shared task/environment/oracle foundation remains a sound bounded goal. The RFC proposes a separate search-policy contract; it should not silently become a prerequisite or expand this tracker's benchmark acceptance. Please revise these three design points before implementing that contract:

P2 — G0 rejects useful incremental candidates (§3.3). upper_bound < residual_target compares one candidate's ceiling with the entire remaining campaign target. In a hypothetical campaign targeting a 10% latency reduction, two independent changes could each remove 6% of the original E2E time and jointly meet the goal, yet G0 refuses both before either runs. Preserve candidate contribution/composition, and reserve the whole-target impossibility test for the combined remaining opportunity. The existing search policy applies its campaign ceiling to all remaining stages together; the campaign template also retains conditional headroom, overlap, queueing, capacity and uncertainty. An unknown or inapplicable bound should remain unverified.
P2 — Static/profile tiers cannot be represented by RunSpec presets alone (§3.4–3.5). static/config supplies only validation, and profile-run explicitly has no A/B pair. Current RunSpec.validate() requires nonempty baseline and candidate commands, and the runner executes paired measurements before optional validation. Define diagnostic/correctness records and their transitions/qualification separately, keeping RunSpec for actual comparisons. Do not fabricate a baseline or performance verdict to pass validation. Clarify how a no-verdict diagnostic can advance under the positive-only escalation rule without pretending it completed P5/P6 comparison.
P2 — S4's per-cell terminal-entry test does not establish search exhaustion (§3.6–3.7). One matrix cell can contain several mechanisms and candidates. One failed candidate in each cell still leaves other candidates untried; G2 itself correctly says a negative hit is conditional evidence, not a veto. Scope exhaustion to an explicitly declared, versioned candidate/hypothesis set and matching context, consistent with the existing candidate-based stopping rule. Retain all terminal records, including correctness failures and inconclusive runs, but keep retention separate from eligibility to exclude a candidate or satisfy S4; inconclusive evidence need not establish a causal search constraint.

Next: @Dong1017 should retain #252's small shared-task scope, identify an accountable owner for the proposed policy contract, and settle these admission/execution/stop semantics before delegating it. Implementation owner and initial benchmark subset remain unassigned; neither this review nor the RFC demonstrates search benefit, accepts a benchmark delivery, or authorizes execution.

当前分工已按用户明确委托收敛为两项工作；#252 继续作为总跟踪。 这次整理不改变原父任务的目标或维护者验收状态。

工作	交付	当前状态
#337：接入现有执行与评测流程	题库/外部项目任务 → 真实 Agent → 实际产物 → 独立验收 → 效果与整轮消耗记录	真实 Agent 的 NPU 链路尚未完成；复用已有 Workbench/SkillEval/执行机制
#336：持续建设优化任务库	每批选择题源、研究采用方案、实现题包和裁判、运行设备验证	首题 RMSNorm 已有工具控制证据；其他类别仍分别研究和适配

原 #334 的研究、#335 的接口已迁入上述工作；两项旧 Issue 按范围合并关闭，不是按功能完成或项目验收关闭。研究与接口文档、历史试点和原始结果继续保留。

首轮计划 2 个 PR：现有 Draft #338 交付共享任务/独立工具/NPU 首题；后续一个 Draft PR（尚未创建）交付 #337 的真实 Agent 接入、外部任务、消耗关联与验收。后续加题按批次在 #336 下提交 PR。

通用流程不强制 profiling，它是 Agent 可选工具。评测层不理解 indexer、Agent 或知识库开关等实验含义，调用方控制变量并重复执行。外部任务以事先固定的项目测试/benchmark 为验收依据，不假设存在万能裁判，也不要求先入题库。

当前任务操作 · CPU 产物交接 · 接入计划与验收 · 任务库计划 · 历史研究、计划与完整实验记录

当前代码交付 6ed93bb41f7db9dafb420dcb6f8705f1864bc45c，PR 已缩减为 39 个文件、+3901/-75 行。源码树不再保留 docs/plans/issue-252/，必要用法改为按功能命名的文档；计划与进度由 Issue 承载，历史材料在上述固定版本中保留。当前内置任务为 rmsnorm-npu 与 softmax-cpu，CPU RMSNorm 已移除。本次整理没有改变评测运行逻辑，也没有改动正在进行的实验所固定的源码。NPU 实测源码仍为 6308268fc6d1fbf90e9ab97b5da8af79152f39ea；历史设备证据未重跑或改标。Canonical-loader finding 已获 scoped clearance；NPU 单次/整体超时 finding仍未解决。参考控制结果不等于 Agent、知识库或服务收益。

#248 的整体效果实验、#249/Optimization Search Policy 的搜索职责保持原归属。本次不接管或重启其他任务。#336、#337、#252 保持 open，#338 保持 Draft；用户已授权本次整理/发布及弃用项关闭，剩余项目验收与合并决定另行记录。原生子项关系此前受写权限限制，继续用引用和此索引跟踪。

标签整理限制：当前账号可以编辑自身 Issue 并关闭旧项，但 GitHub 拒绝了标签增删权限。#334/#335 的实际状态已经是 CLOSED / NOT_PLANNED，原 status:proposed/priority 标签暂留；#336 正文已改为长期跟踪，kind:tracking 标签暂未能替换。该标签差异不改变本次范围迁移及原始证据状态。

Intent: the tracker asks the project to agree and deliver a minimal shared engineering benchmark foundation; the new comment reports a first CPU subset implemented in draft PR #338 and asks maintainers to adopt #334–#337 as native sub-issues. Verdict: human decision required — no accepted scope is recorded and this Issue explicitly withholds implementation authorization, so no development task should start. I verified the PR head's hosted checks pass and the new test module parametrizes to exactly 26 cases, but the raw samples/profiles and the Windows baseline comparison remain local-only and every tracker acceptance item is still open. The earlier design review of the separate search-policy RFC is untouched by this update (the RFC is kept out of this delivery, and the PR's check/measure/profile split matches its point 2), so it is not re-raised. Labels: priority:P2 (tracking umbrella whose decision is pending; no recorded deadline or blocking dependency).

F1 — The proposed delivery precedes the tracker's own scope/interface decision, and its authorization claim is not recorded

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Ownership and authorization" and acceptance checklist items 1–2; AGENTS.md:117; docs/agent-collaboration.md:13-15.
Problem: The update reports a finished 33-file, +1,962-line draft PR (feat: add shared engineering task evaluation and Ascend RMSNorm #338, head d0ce9d785e1bead9124eed1cf3f6e9f2e0cf05db, base 663f3b327998526a13aa31e72ad729025c18983c, author fork) while [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 still has unchecked "Agree a small initial SearchSpace/task subset, upstream reuse choices…" and "Agree the concrete task/environment/oracle interface and scoped delivery criteria before implementation", with no recorded decision. The PR text states "the requester explicitly authorized local implementation and publication on 2026-11-03", but no Issue comment or label records an accepted scope, and the Issue body itself says that reported delegation "is not a recorded acceptance of this parent scope". Assignment identifies accountability, not authority (docs/agent-collaboration.md:13-15).
Conditions: Any decision to keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 moving, to record acceptance on superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337, or to turn this tracker into an implementation task.
Impact: The owner must review a pre-built design instead of deciding the subset/interface first; a rejection discards the work. All four slices still carry the open item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and the tracker cannot close on a draft PR plus green CI.
Fix: Have the named decision owner record one comment accepting, revising or rejecting the CPU subset and the task/oracle interface; keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 draft until that exists, and treat the PR's authorization sentence as a proposal rather than recorded delegation.

F2 — The claimed validation is not reviewable from repository artifacts

Where: docs/plans/issue-252/validation.md and validation-summary.json in PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 at d0ce9d785e1bead9124eed1cf3f6e9f2e0cf05db; repository evidence rules docs/contributing.md:105-107 and evidence/README.md.
Problem: The validation record says the full raw logs, commands, stderr, timing samples and CPU profiles stay in a local directory outside the repository, and that the committed summary cannot substitute for independent replay. The committed summary reports campaign_acceptance: INCONCLUSIVE, 6 trials and 180 measured calls per task (360 total), but the samples, profiles and the claim that 56 Windows failures all reproduce on the unmodified baseline are not in the repo. The acceptance checklist asks for raw samples, rejected changes and evidence paths (docs/contributing.md:105-107).
What I could verify: hosted checks at the PR head are green (full-check 3.10/3.11, installed-package 3.10/3.11, declarations, pr-references; kv-task-qualification skipped), the new test module's 13 test functions parametrize to exactly 26 cases (matching the reported count), and the reuse claim is plausible at import level (harness/benchmarks/cli.py imports harness.runner.run_experiment; consumers.py imports scenarios.external.execute_trial and skill_evaluation.valid_checks). None of the new files exist on main at 663f3b3: no src/inferrsi/harness/benchmarks/, docs/plans/issue-252/ or src/inferrsi/data/engineering_benchmarks/ (verified in the checkout).
Impact: An acceptance owner cannot check sample retention, the baseline comparison or the stated coverage limits from repository bytes, so the tracker criterion "Deliver and validate the selected subset, retaining reproducible measurements" is not yet met; green draft CI establishes CI only.
Fix: Publish the per-task evidence (samples, commands, stderr, profile, hashes) with the PR or make tools/engineering_benchmark_validate.py reproduce the summary from committed inputs, and put the Windows baseline comparison in the same reviewable form.

F3 — The four proposed slices are not children of the tracker, and one slice is outside the pending-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (sub_issues_summary.total = 0), superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337, PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 body, and task: connect engineering tasks to existing Agent execution and evaluation #337's labels.
Problem: The update's tracking exists only as cross-references: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links, and the request to add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 needs a maintainer with write access (the comment reports the sub-issue API returns 404 for a READ-only account). PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 correctly uses non-closing Refs (docs/agent-collaboration.md:102), so merging would close nothing and progress will not roll up. task: connect engineering tasks to existing Agent execution and evaluation #337 also lacks status:proposed and priority:P2 (and bot-reviewed) that superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336 carry, so it does not appear in the same status:proposed decision queue.
Conditions: Deciding whether to adopt this four-slice split and tracking its acceptance.
Impact: Five records (tracker, four slices, one draft PR) must be reconciled by hand; the umbrella cannot show which slice is proposed, accepted or complete, and task: connect engineering tasks to existing Agent execution and evaluation #337 is easy to miss.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels with its siblings. If not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep only the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record which minimal shared engineering benchmark subset, contract and owner are agreed; this edit records that @Bounty-hunter is now the assignee and has proposed four slices delivered together in draft #338, while accept/revise/reject stays with Dong1017. Verdict: human decision required — no accepted scope is recorded and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). The update's checkable claims hold: hosted checks at d0ce9d7 are green (full-check 3.10/3.11, 3570 passed on 3.11, installed-package, pr-references), kv-task-qualification is skipped by its draft == false gate, and validation-summary.json matches the reported INCONCLUSIVE / 360-sample state. F1: open — delivery and assignment are reported, but no accept/revise/reject decision is recorded, and the PR's unlinked "explicitly authorized local implementation" claim is still absent from the record. F2: open — re-read the pinned validation record: raw logs, samples, profiles and the Windows baseline comparison still stay outside the repository, so "retaining reproducible measurements" remains unreviewable from repository bytes (the new focused tests do run in the green hosted Linux suite). F3: open — #252 still has zero native sub-issues and #337 still lacks status:proposed/priority:P2/bot-reviewed. No label value is suggested: every axis already carries a value.

F4 — medium, actionable — The new status block contradicts the unchanged ownership text, so the tracker no longer records one authorization state

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — new "Current state"/"Next action" block vs the unchanged "Ownership and authorization" section, the unchecked acceptance item "Agree a small initial SearchSpace/task subset, upstream reuse choices and accountable implementer", and the closing "Next step"; AGENTS.md:117; docs/agent-collaboration.md:13-17.
Problem: The edit adds an assignee, four proposed slices and a delivered draft PR, but leaves text asserting the opposite: "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The new "superseding the earlier unassigned status below" wording covers only the first of these. The first acceptance item still reads as unmet on the implementer half although an accountable implementer is now recorded, and all four child Issues (superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337) remain unassigned while the parent is assigned.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code") or deciding whether further benchmark work may start or be scheduled.
Impact: The governing record simultaneously answers "implementer assigned, work delivered" and "no implementer, implementation unauthorized and not yet scheduled". docs/agent-collaboration.md:13-17 requires accepted scope and delegation to be recorded or linked, and a self-contradicting record cannot serve as that link; the effect is the same ambiguity that made F1 possible, now inside the Issue itself rather than only in the PR.
Fix: Rewrite "Ownership and authorization" and the closing "Next step" to the recorded state (accountable parent/coordinator @Bounty-hunter; acceptance owner Dong1017; decision pending; child implementers unassigned) and mark or delete the superseded sentences, then split the first checklist item into its recorded half (implementer recorded, done by assignment) and its open half (subset and reuse choices agreed). help:wanted ("Needs an owner or contributor") also now sits next to a recorded assignee; whether to reconcile that label is the owner's call, not something this review applies.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, contract and owner, and it now also records that its first CPU delivery in draft #338 has an unresolved maintainer review finding. Verdict: human decision required — no accept/revise/reject decision is recorded, the body still withholds implementation authorization, and nobody should start a development task. F1: open — the edit documents the unresolved #338 finding but the only decision-shaped event is Dong1017's PR review comment, which accepts nothing on scope. F2: open — raw samples/profiles remain outside the repository and the new body sentence confirms it, so "retaining reproducible measurements" is still unreviewable, while the cited timing samples come from the loop flagged at numpy_task.py:125. F3: open — #252 still has zero native sub-issues and #337 still lacks status:proposed/priority:P2. F4: open — the new "Next action" now directs work on #338 while the unchanged "Ownership and authorization" and "Next step" say implementation is not authorized and must wait. No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit records delivery and an unresolved review, but still no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; fork plan docs/plans/issue-252/README.md at d0ce9d7; docs/agent-collaboration.md:13-17; AGENTS.md:117.
Problem: The updated block adds the unresolved feat: add shared engineering task evaluation and Ascend RMSNorm #338 timing finding to the state, but no decision from the acceptance owner is recorded. Verified: the only decision-shaped event since the last review is Dong1017's PR review comment r4456416093 (2026-11-03T15:00:54Z) on src/inferrsi/harness/benchmarks/numpy_task.py:125, which reviews measurement code and does not accept the CPU subset, the task/oracle contract, or the four slices. The fork plan still rests on a "user's explicit delegation on 2026-11-03" that has no counterpart anywhere in this repository's record, and the body itself concedes that delegation "is not a recorded acceptance of this parent scope".
Conditions: Any move to keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 moving toward merges, to mark acceptance on superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337, or to treat this tracker as an authorized implementation task.
Impact: Work continues ahead of the tracker's own decision; all four children still carry the unchecked item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and a rejection discards a 33-file, +1,962-line delivery.
Fix: Have the named decision owner post one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 draft until that exists. A review comment on the PR is not that decision.

F2 — high, human decision required — The validation evidence is still not reviewable, and the cited timing samples come from the path a maintainer review says can measure replay

Where: docs/plans/issue-252/validation.md at d0ce9d7 (raw run kept in an out-of-repository inferrsi-252-validation-final directory; states the public summary "不能代替独立重放"), the new body sentence about unpublished raw timing/profile logs, and the open finding at src/inferrsi/harness/benchmarks/numpy_task.py:125.
Problem: Acceptance item 3 asks to "validate the selected subset, retaining reproducible measurements". The 360 samples offered as the delivery state were produced by the loop r4456416093 flags: the precheck calls compute(x) and every warmup/sample then reuses the same x, so a process-local memoizing candidate can satisfy the value assertions while most timed calls only replay stored output. Because samples and profiles stay outside the repository, an acceptance owner cannot tell measured execution from replay, nor re-check the "56 Windows failures reproduce on baseline" claim.
Conditions: Any decision that rests on suite_latency_ms, on the reported profile, or on the Windows baseline comparison.
Impact: "Reproducible measurements" cannot be confirmed from repository bytes, and a timing-path correctness defect is now recorded as known but not tracked anywhere in the acceptance checklist; green hosted checks establish CI only (full-check 3.10/3.11, installed-package 3.10/3.11, pr-references pass; kv-task-qualification skipped).
Fix: Publish per-task samples, commands, stderr, profiles and hashes, or make tools/engineering_benchmark_validate.py rebuild the summary from committed inputs; fix the timing boundary (frozen varying input sequence shared by baseline and candidate, plus a stateful memoizing negative regression) and republish before citing any latency number.

F3 — medium, human decision required — The status block still contradicts the unchanged ownership and next-step text, now including a direct work instruction

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — new "Current state"/"Next action" block vs the unchanged "Ownership and authorization" section, the closing "Next step", and acceptance item 1; docs/agent-collaboration.md:13-17.
Problem: The new "Next action" tells the assignee to "address the unresolved feat: add shared engineering task evaluation and Ascend RMSNorm #338 review finding with current-head evidence", while the body still says "Current authorization is to track and clarify this work, not to launch implementation", "Implementer and initial subset remain unassigned", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The "superseding the earlier unassigned status below" wording still covers only the assignee half.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code") or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record simultaneously answers "assignee coordinating delivery work" and "no implementer, implementation unauthorized, not yet scheduled"; that ambiguity is what let the pre-built delivery arise in the first place.
Fix: Rewrite "Ownership and authorization" and "Next step" to the recorded state (accountable coordinator @Bounty-hunter, acceptance owner Dong1017, decision pending, child implementers unassigned), delete or mark the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half (subset and reuse choices agreed).

F4 — medium, human decision required — The four slices are still not children of the tracker, and #337 is still outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (sub_issues returns an empty list), superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 labels, PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues; superseded: benchmark research moved to task library and integration #334, superseded: task contracts moved to evaluation integration #335 and tracking: grow reusable engineering optimization tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect engineering tasks to existing Agent execution and evaluation #337 carries only kind:task and area:workbench. PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 uses non-closing references, so merge progress would roll up nowhere. The new task: connect engineering tasks to existing Agent execution and evaluation #337 preregistration comment (2026-11-03T15:29:12Z) — the newest evidence the body relies on — therefore lands on an Issue that does not appear in the same status:proposed decision queue as its siblings.
Conditions: Deciding whether to adopt the four-slice split and tracking its acceptance.
Impact: Five records must be reconciled by hand, and the tracker cannot show which slice is proposed, accepted or complete; task: connect engineering tasks to existing Agent execution and evaluation #337 is easy to miss.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels with its siblings. If not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, task/oracle contract and delivery criteria; this edit records Bounty-hunter as assignee, pins draft #338 at c06b725 and reports a new 720-sample usefulness supplement. Verdict: human decision required — no accept/revise/reject decision exists and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). The edit's checkable claims hold: usefulness-summary.json contains exactly 720 samples, two profiles are published, no-epsilon passes the ordinary smoke but fails six assertions, both comparisons are FAIL (-50.52% unchanged-vs-unchanged, -76.95% einsum), and run 41665826142 is green with kv-task-qualification skipped. F1: open. F2: partially addressed — samples, profiles and hashes are now in-tree, but command logs/stderr/source snapshots remain local and the samples predate the head's v2 input protocol. F3: open. F4: open (sub_issues is still empty; #337 still carries only kind:task and area:workbench). F5 (new): the state block pins a superseded PR head.

F1 — high, human decision required — The edit still records delivery and an unresolved review, but no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117; docs/agent-collaboration.md:14-17.
Problem: The update records an assignee, four proposed slices and draft feat: add shared engineering task evaluation and Ascend RMSNorm #338 at c06b7255, and states the decision is pending, but no decision from the acceptance owner exists. The only decision-shaped event since the last round remains Dong1017's PR review comment r4456416093 on src/inferrsi/harness/benchmarks/numpy_task.py:125, which reviews measurement code and accepts nothing about the CPU subset, the task/oracle contract or the four slices. The body itself concedes the reported local delegation "is not a recorded acceptance of this parent scope". Assignment identifies accountability, not authority (docs/agent-collaboration.md:14), and work is supposed to follow accepted scope or recorded delegation (docs/agent-collaboration.md:15).
Conditions: Any move to keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 moving toward merge, mark acceptance on superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337, or treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: Work continues ahead of the tracker's own decision; all four children still carry the open item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and a rejection discards the delivery.
Fix: Have Dong1017 post one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 draft until that exists.

Status: open (unchanged this round).

F2 — high, human decision required — Evidence retention improved, but the 720 samples are v1 and the run commands still are not reviewable

Where: docs/plans/issue-252/usefulness.md and usefulness-summary.json at c06b7255; docs/plans/issue-252/validation.md; [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 acceptance item 3.
Problem: The edit's claim is accurate: usefulness-summary.json holds exactly 720 samples_ms values across 144 arrays, the two usefulness-profiles/*.txt files exist, and the no-epsilon/einsum results match. What remains unreviewable is the rest: the usefulness.md reproduction section still says "完整原始日志留在本地" and the validation.md initial run (26 focused tests, 360 samples, 56 Windows failures "all reproduced on the unmodified baseline") is still summary-only, so the baseline comparison and command/stderr evidence cannot be checked from repository bytes. In addition, the published samples were produced by the v1 precheck/warmup/measure loop the P2 review flagged, and the PR head has since replaced that protocol, so they do not establish reproducibility for the current runtime.
Conditions: Any acceptance decision that rests on the 720 samples, the profiles or the Windows baseline comparison.
Impact: Acceptance item 3 ("retaining reproducible measurements") is only partly met; a decision owner still cannot distinguish measured execution from replay for the initial 360-sample record, and the newly published numbers are tied to a superseded task version.
Fix: Publish the remaining commands/stderr/source-snapshot hashes (or make tools/engineering_benchmark_validate.py rebuild the summary from committed inputs), and label the 720-sample supplement explicitly as v1 so it is not read as evidence for the current head.

Status: partially addressed (samples, profiles and hashes are now in-tree; commands/logs and the current-runtime rerun are missing).

F5 — medium, actionable — The state block pins a superseded PR head and describes a fixed timing path as unresolved

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Current state" ("at c06b72555d4643df133c59fe7cead5a91e7b86f5 (runtime/task definitions unchanged from d0ce9d7...)", "An unresolved current-head P2 review …", "Current-head hosted full-suite/install checks … passed").
Problem: The PR head is now ce84506e7106c4fc48fafdfab55d2ea2f9dffd50 ("fix: bind benchmark timing to distinct paired inputs", committed 2026-11-03T16:04:22Z, after the body edit at 16:03:29Z). That commit changes src/inferrsi/harness/benchmarks/numpy_task.py (+73/−30) plus both task.json/README files to the numpy-axis1-v2 protocol, adds replay-validation.md, replay-validation-summary.json and +123 lines of tests, and its own record says "任务、grader 身份和指标的五 case 聚合均已变化". At head, evaluate() now builds per-call inputs with inputs(case, np, sequence=index + 1) and rejects repeated input bytes, which is the repair the P2 finding asked for. So the three statements above no longer describe the PR: runtime/task definitions did change, the P2 is addressed by an unreviewed fix commit rather than open at head, and run 41665826142's green checks belong to c06b7255 — at ce84506e full-check (3.10/3.11) was still in progress when checked.
Conditions: Any decision owner reading the tracker to evaluate the delivery state or the evidence behind it.
Impact: The governing record would have acceptance decided against a superseded measurement protocol and stale CI status; the published 720 samples are v1 and do not automatically validate the v2 runtime that is now at the PR head.
Fix: Update the block to the actual head, say whether the v2 fix is reviewed and which evidence/suite status belongs to it, and keep the v1 supplement labeled as v1 (or wait for the head's checks and review before re-stamping the state).

F3 — medium, actionable — The status block still contradicts the unchanged ownership and next-step text

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — new "Current state"/"Next action" block vs the unchanged "Ownership and authorization" section, the closing "Next step", and acceptance item 1; AGENTS.md:117.
Problem: The edit adds a new "Next action" telling the assignee to coordinate the scope decision and "address the unresolved feat: add shared engineering task evaluation and Ascend RMSNorm #338 review finding with current-head evidence", while the body below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation, claim another contributor's work, run paid models or allocate hardware", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The "superseding the earlier unassigned status below" wording still covers only the assignee half.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code") or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record simultaneously answers "assignee coordinating delivery work" and "no implementer, implementation unauthorized, not yet scheduled", which is the ambiguity that allowed the pre-built delivery.
Fix: Rewrite "Ownership and authorization" and "Next step" to the recorded state (accountable coordinator Bounty-hunter, acceptance owner Dong1017, decision pending, child implementers unassigned), mark or delete the superseded sentences, and split acceptance item 1 into its recorded half and its open half.

Status: open (the new "Next action" adds a work instruction but leaves the contradictory text in place).

F4 — medium, actionable — The four slices are still not children of the tracker, and #337 remains outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns an empty array), superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 labels, PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 still has no native sub-issues or dependency links. superseded: benchmark research moved to task library and integration #334, superseded: task contracts moved to evaluation integration #335 and tracking: grow reusable engineering optimization tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect engineering tasks to existing Agent execution and evaluation #337 carries only kind:task and area:workbench, so the Issue that now holds the supplemental results comment (6608126902) does not appear in the same status:proposed decision queue as its siblings. PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 uses non-closing references, so merge progress would roll up nowhere.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records must be reconciled by hand and the tracker cannot show which slice is proposed, accepted or complete; task: connect engineering tasks to existing Agent execution and evaluation #337 is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Status: open (verified unchanged this round).

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, task/oracle contract and owner; this edit re-pins the delivery to draft #338 at ac03ae2, reports the numpy-axis1-v2 replay fix, and cites the v2 record (40 author-reported tests, 300 timing samples) plus green hosted checks. Verdict: human decision required — the edit's checkable claims hold, but no accept/revise/reject decision is recorded and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). F1: open. F2: partially addressed (v2 identity hashes and planned/executed match flags are in-tree, but no timing values, profiles or commands are, so the 300 samples remain local-only). F3: open. F4: open (sub_issues is still empty; #337 still carries only kind:task and area:workbench). F5: fixed — the body now pins ac03ae2, the actual draft head; run 41672237031 has that head SHA with all checks green/skipped, and GraphQL confirms the P2 thread isOutdated=true/isResolved=false, matching the new wording. No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit still records delivery and an unresolved review, but no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; docs/agent-collaboration.md:13-17; AGENTS.md:117.
Problem: The status block was rewritten again and now points at ac03ae2, but no decision from the acceptance owner is recorded. The only decision-shaped events remain Dong1017's PR review comment r4456416093 (a P2 measurement finding on numpy_task.py) and the author's reply r4457228612, which itself says it does not mark the thread resolved and makes no maintainer-acceptance claim. The body concedes the reported local delegation "is not recorded parent scope acceptance".
Conditions: Any move to keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 moving toward merge, to mark acceptance on superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337, or to treat this tracker as an authorized implementation task.
Impact: Work continues ahead of the tracker's own decision; all four children still carry the unchecked item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope and a rejection discards the delivery.
Fix: Dong1017 posts one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 draft until that exists. A review comment on the PR is not that decision.

F2 — high, human decision required — The v2 record publishes identities and input hashes but still no measurements

Where: docs/plans/issue-252/replay-validation.md:42 and docs/plans/issue-252/replay-validation-summary.json at ac03ae2; [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 acceptance item 3 ("retaining reproducible measurements").
Problem: Verified this round: the committed v2 summary carries identity hashes, correctness/verdict fields, measurement_trials: 6, measured_calls: 150 per task, all_workloads_match/all_input_sequences_match/all_valid_executed_calls_match: true and a 35-entry input_sequence with per-call SHA-256, but no samples_ms, median or latency field at all; replay-validation.md:42 states the raw timings, logs and profiles remain local. The superseded v1 usefulness supplement published 720 samples and two profiles in-tree, so the current-head record is less reviewable than the evidence it replaces, and the "40 focused tests" / "300 timing samples" figures are author-reported.
Conditions: Any acceptance decision that rests on the v2 record or on the word "validated".
Impact: Acceptance item 3 cannot be checked from repository bytes, and the decision owner cannot re-derive the summary or inspect the timing path the P2 thread questioned; green hosted checks establish CI only.
Fix: Publish per-call samples_ms (as usefulness-summary.json already does) or make tools/engineering_benchmark_validate.py regenerate the summary from committed inputs, and say which figures are v2 versus historical v1.

F6 — medium, human decision required — The new state block drops the recorded local Windows suite failure

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Current state" (the sentence now reading "The v1 360-sample integration and separate 720-sample usefulness results remain historical evidence…" plus "Current-head hosted full-suite/install/reference checks passed"), against docs/plans/issue-252/validation.md:40-41 at ac03ae2.
Problem: The previous body recorded that the full Windows suite remains non-green, with the failures reproduced on the unmodified baseline. The edit keeps the v1 samples as "historical evidence" but drops the suite status, and its remaining-gaps sentence lists same-code noise, performance ordering, SkillEval two-arm, agent/skill benefit and hardware — not the suite. The v2 record itself reports only the 40 focused tests plus CLI/SkillEval regressions, so no full-suite run is recorded at ac03ae2; validation.md:40-41 still says "3479 passed, 51 skipped, 56 failed … 全部在未修改的原始基线复现 … 全量结果仍为非绿色".
Conditions: An acceptance decision that reads the green hosted run (run 41672237031) as suite-clean for the delivery.
Impact: Acceptance item 5 explicitly says not to close because software CI is green; removing the caveat from the governing record makes exactly that misreading easier and weakens the "preserve failures" commitment in the previous "Next action".
Fix: Add one sentence to the state block: the local Windows suite remains non-green (56 failures, all reproduced on the unmodified baseline), and v2 has no recorded full-suite run beyond hosted Linux CI.

F3 — medium, human decision required — The status block still contradicts the unchanged ownership and next-step text

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — new "Current state"/"Next action" block vs the unchanged "Ownership and authorization" section, the closing "Next step", and acceptance item 1; AGENTS.md:117.
Problem: The edit adds "Existing assignee @Bounty-hunter coordinates the recorded scope decision and current-head re-review", while the body below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". Nothing in the new block marks those sentences superseded, although an implementer is recorded and the work has been delivered.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code") or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record simultaneously answers "assignee coordinating delivery and re-review" and "no implementer, implementation unauthorized, not yet scheduled", which is the ambiguity that allowed the pre-built delivery.
Fix: Rewrite "Ownership and authorization" and "Next step" to the recorded state (accountable coordinator Bounty-hunter, acceptance owner Dong1017, decision pending, child implementers unassigned), mark the superseded sentences, and split acceptance item 1 into its recorded half and its open half (subset and reuse choices agreed).

F4 — medium, human decision required — The four slices are still not children of the tracker, and #337 remains outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns an empty array), superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 labels, PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 still has no native sub-issues. superseded: benchmark research moved to task library and integration #334, superseded: task contracts moved to evaluation integration #335 and tracking: grow reusable engineering optimization tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect engineering tasks to existing Agent execution and evaluation #337 (updated 2026-11-03T16:16:09Z, i.e. after the v2 work) still carries only kind:task and area:workbench. PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 uses non-closing references, so merge progress would roll up nowhere, and the body's new sentence that child Issues mix v1 and current-head links has no rollup surface to correct.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records must be reconciled by hand and the tracker cannot show which slice is proposed, accepted or complete; task: connect engineering tasks to existing Agent execution and evaluation #337 is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, task/oracle contract and delivery criteria; this edit re-pins draft #338 at ac03ae23, cites Dong1017's scoped source confirmation (r4459426716) that the cached-output replay finding is addressed, and asks for a full current-head PR review. Verdict: human decision required — the edit's checkable claims hold (PR head and run 41672237031 both at ac03ae23, thread still outdated/unresolved, v2 record still local-only timings), but no accept/revise/reject decision is recorded and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). F1: open. F2: open. F6: open. F3: open. F4: open (verified: #252 sub-issues still empty, #337 still lacks status:proposed/priority:P2). F5: remains fixed. No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit records a scoped technical confirmation, still no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117; docs/agent-collaboration.md:14-16.
Problem: The new sentence cites r4459426716 as confirming the replay fix, and that comment is accurate about its own limits — it was written by the acceptance owner Dong1017 at 2026-11-03T19:23:55Z and says "Full-PR review, v2 timing/noise qualification and the recorded parent scope decision remain open". So the only new decision-shaped event is explicitly not the scope decision. Checks this round: no acceptance comment exists on superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 (their newest comments are the reporter's on task: connect engineering tasks to existing Agent execution and evaluation #337 and bot reviews on superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336), and the body itself concedes the reported local delegation "is not recorded parent scope acceptance". Assignment and a scoped review identify accountability, not accepted scope (docs/agent-collaboration.md:14-16).
Conditions: Any move to keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 moving toward merge, to mark acceptance on superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337, or to treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: All four children still carry the unchecked item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and a rejection discards the delivery.
Fix: Dong1017 posts one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 draft until that exists. A scoped PR review comment is not that decision.

Status: open (unchanged in substance; the 19:23 comment narrows one technical point only).

F2 — high, human decision required — The v2 record is still less reviewable than the v1 evidence it supersedes

Where: docs/plans/issue-252/replay-validation.md ("完整原始计时、日志和 profile 仍在本地") and docs/plans/issue-252/replay-validation-summary.json at ac03ae23; acceptance item 3 ("retaining reproducible measurements").
Problem: Re-verified at the pinned head: the committed v2 summary contains identity hashes and per-input digests but no samples_ms, median or latency field at all, and the record states raw timings, logs and profiles stay local. The superseded v1 usefulness supplement published 720 samples and two profiles in-tree, so the current-head record is the least reviewable artifact in the set; the "40 focused tests" and "300 timing samples" figures remain author-reported. The edit states this limitation honestly, but stating it does not meet the acceptance criterion.
Conditions: Any acceptance decision that rests on the v2 record or on the word "validated".
Impact: The decision owner cannot re-derive the summary or inspect the timing path from repository bytes; green hosted checks establish CI only.
Fix: Publish per-call samples_ms (as usefulness-summary.json already does) or make tools/engineering_benchmark_validate.py regenerate the summary from committed inputs, and label every figure as v2 or historical v1.

Status: open (no change this round).

F6 — medium, human decision required — The state block still omits the recorded non-green local suite

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Current state" ("Current-head hosted full-suite/install/reference checks passed") versus docs/plans/issue-252/validation.md:40-41 at ac03ae23.
Problem: The record still reads "3479 passed、51 skipped、56 failed … 全部在未修改的原始基线复现 … 全量结果仍为非绿色，不能称为所有检查通过". The edit keeps the v1 samples as "historical evidence" and preserves "both v1 usefulness performance FAILs", but its remaining-gaps sentence lists only same-code noise, performance ordering, SkillEval two-arm, agent/skill benefit and hardware — not the suite. The green hosted run cited (41672237031) is Linux CI, not the local Windows suite.
Conditions: An acceptance decision that reads the green hosted run as suite-clean for the delivery.
Impact: Acceptance item 5 explicitly forbids closing because software CI is green; removing the caveat from the governing record makes exactly that misreading easier.
Fix: Add one sentence to the state block: the local Windows suite remains non-green (56 failures, all reproduced on the unmodified baseline), and v2 has no recorded full-suite run beyond hosted Linux CI.

Status: open (the edit did not restore the caveat).

F3 — medium, human decision required — The status block still contradicts the unchanged ownership and next-step text

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — "Current state"/"Next action" versus the unchanged "Ownership and authorization" section (AGENTS.md:117 routing rule).
Problem: The new "Next action" tells the assignee to coordinate "full current-head PR review", while the body below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The earlier "superseding the earlier unassigned status below" wording is now gone entirely, so nothing marks those sentences as superseded even though an assignee and a delivered draft are recorded.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code"), or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record simultaneously answers "assignee coordinating delivery and full review" and "no implementer, implementation unauthorized, not yet scheduled" — the ambiguity that allowed the pre-built delivery to arise.
Fix: Rewrite "Ownership and authorization" and "Next step" to the recorded state (accountable coordinator Bounty-hunter, acceptance owner Dong1017, decision pending, child implementers unassigned), mark or delete the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded) and its open half (subset and reuse choices agreed).

Status: open (worse this round: the superseding sentence was removed).

F4 — medium, human decision required — The four slices are still not children of the tracker, and #337 remains outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []), superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 labels, PR feat: add shared engineering task evaluation and Ascend RMSNorm #338.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 still has no native sub-issues. superseded: benchmark research moved to task library and integration #334, superseded: task contracts moved to evaluation integration #335 and tracking: grow reusable engineering optimization tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect engineering tasks to existing Agent execution and evaluation #337 (last updated 2026-11-03T16:16:09Z) still carries only kind:task and area:workbench. PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 uses non-closing references, so merge progress would roll up nowhere, and the body's own sentence that child Issues mix v1 and current-head links has no rollup surface to correct.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records must be reconciled by hand; the tracker cannot show which slice is proposed, accepted or complete, and task: connect engineering tasks to existing Agent execution and evaluation #337 — which holds the supplemental usefulness results — is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Status: open (verified unchanged this round).

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared CPU benchmark subset, task/oracle contract and delivery criteria; this edit re-pins draft #338 at 9454672f, reports the compare exit-status fix and now publishes the 300 timing samples plus two profiles. Verdict: human decision required — the edit's checkable claims hold (PR head equals the pin, run 41739933972 is green on 3.10/3.11 with qualification skipped, benchmarks/ grader digests recompute to 0b75b5bb… at ac03ae23 and 34928e69… at head with only cli.py differing, the export holds 12 trials/300 samples whose per-case medians sum exactly to the recorded suite latencies, and the benchmark test module expands to exactly 43 cases), but no accept/revise/reject decision is recorded, so no development task should start (the Issue is also assigned). F1: open. F2: fixed — per-call samples and profiles are now in-tree and internally consistent. F6: fixed — the state block again names the non-green historical Windows suite. F5: remains fixed. F3: open. F4: open (sub_issues still empty; #337 still lacks status:proposed/priority:P2/bot-reviewed). No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit still records delivery and a scoped confirmation, not an accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117; docs/agent-collaboration.md:13-16.
Problem: The block was rewritten again (exit-status fix, published samples) but still contains no decision from the acceptance owner. Checked this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no comment newer than the bot review; PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 has exactly three reviews — r4456416093 (2026-11-03T15:00Z, the P2 measurement finding), the author reply r4457228612, and r4459426716 (19:23Z) whose own text says "Full-PR review, v2 timing/noise qualification and the recorded parent scope decision remain open"; and superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 still carry no acceptance comment. The body itself concedes the reported local delegation "is not recorded parent scope acceptance", and assignment identifies accountability, not authority (docs/agent-collaboration.md:14).
Conditions: Any move to keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 moving toward merge, mark acceptance on superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337, or treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: All four children still carry the unchecked acceptance item, so nothing can merge as accepted scope and a rejection discards the delivery; the decision owner is being asked to judge a pre-built design instead of deciding subset and interface first.
Fix: Dong1017 posts one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 draft until that exists. Neither a scoped PR review nor an assignee substitutes for it.

F7 — medium, human decision required — The published 300-sample record no longer binds to the head, because the grader digest covers the benchmark CLI

Where: src/inferrsi/harness/benchmarks/contract.py:33-34 and worker.py identity check at 9454672f; docs/plans/issue-252/replay-validation.md:17; [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 acceptance item 3 ("retaining reproducible measurements").
Problem: grader_digest() is tree_digest() of the whole harness/benchmarks/ directory, so the exit-status fix — a 2-line change in cli.py that only adds the retained status to the command output — rotated the grader identity. Recomputed from both trees: ac03ae23 → 0b75b5bb…, 9454672f → 34928e69…, and cli.py is the only differing file. worker.py rejects any identity mismatch, so the just-published replay-measurements.json (all 12 trials carry grader_sha256: 0b75b5bb…) and the two profiles are evidence for the superseded identity only; the current head has no measurement evidence bound to its own identity. The record states this boundary honestly and says the compute/oracle/timing files are unchanged, so this is not a concealment — but acceptance item 3 then rests on the owner accepting that stated equivalence rather than on a head-bound record.
Conditions: Any acceptance decision that cites the v2 record as current-head evidence, and every future change to benchmark CLI/reporting code.
Impact: Measurement identity is coarser than measurement semantics: any reporting or plumbing touch invalidates all prior timing evidence and forces either a re-run or a fresh equivalence argument, which will keep recurring as the CLI evolves.
Fix: Decide and record one of: (a) narrow grader_digest() to the measurement-bearing modules so reporting changes do not rotate the identity, or (b) re-run the v2 validation at the head (tools/engineering_benchmark_validate.py, CPU-only, minutes) so the accepted scope has head-bound samples. Either way, label the 0b75b5bb samples as pre-fix evidence in the acceptance record.

F3 — medium — "Ownership and authorization" and the closing "Next step" still contradict the recorded state

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — the unchanged "Ownership and authorization" section and closing "Next step" versus the new "Current state"/"Next action"; AGENTS.md:117.
Problem: The section still says "Implementer and initial subset remain unassigned" although an assignee is recorded and an implementation has been delivered, and its closing "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation" contradicts a "Next action" that tasks the assignee with coordinating current-head review of that delivery. The new sentence "the original tracker authorization below is unchanged" now points at those sentences rather than reconciling them, so a reader or agent routing per AGENTS.md:117 still gets two answers for "is there an implementer and may this work be scheduled".
Conditions: Any reader or agent using this Issue as the governing record for benchmark work.
Impact: The tracker cannot be cited as one consistent authorization state, which is the ambiguity that allowed the pre-built delivery to arise.
Fix: Rewrite the section and the closing step to the recorded state (accountable assignee/coordinator @Bounty-hunter, acceptance owner Dong1017, scope decision pending, initial subset and reuse choices not yet agreed), and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half.

F4 — medium — The four slices are still not children of the tracker, and #337 is still outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []), superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 labels, PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links. superseded: benchmark research moved to task library and integration #334, superseded: task contracts moved to evaluation integration #335 and tracking: grow reusable engineering optimization tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect engineering tasks to existing Agent execution and evaluation #337 (last updated 2026-11-03T16:16:09Z) carries only kind:task and area:workbench. PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 uses non-closing references, so merge progress rolls up nowhere, and the body's own sentence that child Issues mix v1 and current-head links has no rollup surface to correct.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records (tracker, four children, one draft PR) must be reconciled by hand; the tracker cannot show which slice is proposed, accepted or complete, and task: connect engineering tasks to existing Agent execution and evaluation #337 is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared CPU engineering-benchmark subset, task/oracle contract and delivery criteria; this edit re-pins draft #338 at fd0f396c, adds the engineering-benchmark-v1 correctness-artifact adapter and a 49-test software-control record. Verdict: human decision required — no accept/revise/reject decision exists and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). Verified this round: the pin equals the PR head, runs 41745549558 and 41746656869 match their claims (the later 3.10 check has since finished green), the record's 49 tests/0 skips and 3,631 passed/18 skipped per suite match, and the tested-head-to-head delta is three docs files; F1 open, F7 open (the published samples still carry the pre-fix grader identity), F3/F4 open, F8 is new, and F2/F5/F6 remain fixed. No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit still records a delivered adapter and a green software-control run, but no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code"); docs/agent-collaboration.md:13-16 ("Work follows accepted scope and explicit or standing delegation. Record or link that authority").
Problem: The block was rewritten again (new adapter, 49-test record, new pins), but still contains no decision from the acceptance owner. Checked this round: the only events since the previous review are this body edit and the author's index comment edit at 00:25:49Z (status only); PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 still has exactly three reviews — r4456416093, the author reply r4457228612, and r4459426716, whose own text says the recorded parent scope decision remains open; and superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 still carry no acceptance comment. The body itself concedes that contributor-reported delegation "is not recorded parent scope acceptance".
Conditions: Any move to keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 moving toward merge, to mark acceptance on superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337, or to treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: All four children still carry the unchecked item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and a rejection discards a delivery that keeps growing (the head is now nine commits).
Fix: Dong1017 posts one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 draft until that exists. A scoped PR review, an assignee or a green CI run does not substitute for it.

Status: open (unchanged in substance; the new adapter is technical progress, not a scope decision).

F7 — medium, human decision required — The published timing samples still belong to the pre-exit-fix grader identity, and neither offered fix is chosen

Where: docs/plans/issue-252/replay-measurements.json and replay-validation.md:43 at fd0f396c; src/inferrsi/harness/benchmarks/contract.py:33-34 (grader_digest() = tree_digest() of the whole harness/benchmarks/ directory); [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 acceptance item 3 ("retaining reproducible measurements").
Problem: Re-verified at the current head: all 12 published trials still carry grader_sha256: 0b75b5bb… (the ac03ae23 identity), while recomputing tree_digest() over src/inferrsi/harness/benchmarks/ at fd0f396c yields 34928e69… — unchanged since 9454672f, because the new adapter lives in src/inferrsi/harness/skill_eval_benchmark.py, outside the digested tree. The new adapter now freezes grader_sha256 = 34928e69… in every task descriptor, so the published samples and profiles are evidence for neither the current head nor the adapter's own frozen identity. The edit labels them "historical and not rebound to this adapter/head", which addresses the labeling half of the earlier finding, but neither remedy (narrow the digest or re-run at the head) is decided.
Conditions: Any acceptance decision that cites the v2 record as current-head evidence, and every future change to benchmark CLI/reporting code.
Impact: Acceptance item 3 still rests on the owner accepting a stated equivalence argument rather than on a head-bound record, and each reporting/plumbing change rotates the identity and invalidates prior timing evidence again.
Fix: Record one decision: (a) narrow grader_digest() to the measurement-bearing modules so reporting changes do not rotate identity, or (b) re-run tools/engineering_benchmark_validate.py (CPU-only, minutes) at fd0f396c and publish head-bound samples. Keep the 0b75b5bb export labelled as pre-exit-fix evidence either way.

Status: open (acknowledged and labelled; no fix chosen).

F8 — medium, human decision required — The same state block says formal SkillEval compare integration is both delivered and incomplete

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Current state" (first paragraph: the adapter in skill_eval_compare/skill_eval_local "supersedes the earlier unimplemented formal-compare status"; later gap sentence: "formal SkillEval compare/improve integration … remain incomplete"); docs/plans/issue-252/README.md ("第二条现已增加 skill_eval_compare 的正式产物交接"); docs/plans/issue-252/04-integration.md (status: 正式 SkillEval 正确性产物交接已实现); docs/plans/issue-252/replay-validation.md:52 ("正式 SkillEval 两臂产物交接 … 仍未完成").
Problem: The edit adds a delivered compare/local adapter and explicitly says it supersedes the earlier unimplemented formal-compare status, then lists "formal SkillEval compare/improve integration" as still incomplete. The plan docs support the first reading (compare implemented, skill_eval_improve not), while replay-validation.md:52 still says the two-arm artifact handoff is outstanding. Only skill_eval_improve and real agent-produced two-arm experiments are actually absent (formal-integration.md says provider generation, skill_eval_improve and real Agent runs are not added).
Conditions: The acceptance owner reading the tracker to decide acceptance item 4 ("Demonstrate that the selected shared package can be consumed by its agreed Workbench/evaluation use cases…").
Impact: The decision owner cannot tell whether the SkillEval half of acceptance item 4 is delivered at the software-control level or still missing, which invites either accepting without knowing what exists or re-requesting work already done.
Fix: Replace the compound sentence with the recorded split — engineering-benchmark-v1 compare/local adapter implemented and covered by the 49-test control; real two-arm agent runs, provider generation, skill_eval_improve and reliable ranking still absent — and update the stale sentence at replay-validation.md:52.

Status: new this round.

F3 — medium — "Ownership and authorization" and the closing "Next step" still contradict the recorded state

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — unchanged "Ownership and authorization" section ("Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation") versus the new "Current state"/"Next action" block, which says "the original tracker authorization below is unchanged" and tasks the assignee with coordinating the scope decision and full current-head review of a delivered implementation; acceptance item 1; AGENTS.md:117.
Problem: The new sentence now explicitly points at the contradictory paragraphs instead of reconciling them. An implementer and a delivered draft are recorded, yet the governing text still says no implementer exists and implementation is unauthorized and unscheduled.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117, or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record gives two answers to "is there an implementer and may this work be scheduled" — the ambiguity that allowed the pre-built delivery to arise.
Fix: Rewrite the section and closing step to the recorded state (accountable coordinator @Bounty-hunter, acceptance owner Dong1017, scope decision pending, initial subset and reuse choices not yet agreed), delete or mark the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half.

Status: open (unchanged; the new "unchanged" sentence makes the conflict explicit).

F4 — medium — The four slices are still not children of the tracker, and #337 remains outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []), superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 labels, PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links. superseded: benchmark research moved to task library and integration #334, superseded: task contracts moved to evaluation integration #335 and tracking: grow reusable engineering optimization tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect engineering tasks to existing Agent execution and evaluation #337 (updated 2026-11-04T00:25:47Z) carries only kind:task and area:workbench. PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 uses non-closing references, so merge progress rolls up nowhere.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records (tracker, four children, one draft PR) must be reconciled by hand; the tracker cannot show which slice is proposed, accepted or complete, and task: connect engineering tasks to existing Agent execution and evaluation #337 — which holds the supplemental results — is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Status: open (verified unchanged this round).

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, task/oracle contract and delivery criteria, with Dong1017 owning the accept/revise/reject decision. Verdict: human decision required — this edit's checkable claims all hold (pin at fd0f396, run 41746656869 green with only draft KV qualification skipped, the retained JUnit artifact has 49 tests/0 skips and SHA-256 ac2d2164… as recorded, both full-suite job logs show 3,631 passed/18 skipped, and the dfcc1855→head delta is the three documentation/evidence files it names), but no decision is recorded and the Issue still withholds implementation authorization, so no development task should start. Earlier findings: F2, F5 and F6 fixed; F7 answered by explicitly labelling the 300 v2 samples as historical (head grader digest is 34928e69… versus 0b75b5bb… in those samples); F1, F3 and F4 remain open; I found no new factual defect in this edit.

F1 — high, human decision required — The edit adds verified delivery evidence but still no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117; docs/agent-collaboration.md:14-15; PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 review r4459426716.
Problem: Checked this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no comment newer than the 2026-11-04T00:07Z bot review; PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 has exactly three reviews, the newest being Dong1017's own scoped confirmation (r4459426716, 2026-11-03T19:23Z), which says "Full-PR review, v2 timing/noise qualification and the recorded parent scope decision remain open"; superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 carry no acceptance comment. The edit pins the real head and adds new, verified correctness evidence (49 tests/0 skips bound to dfcc1855, JUnit SHA-256 ac2d2164… matching the retained artifact, docs-only delta to head), but that is implementation progress, not the decision.
Conditions: Any move to merge or advance feat: add shared engineering task evaluation and Ascend RMSNorm #338, mark child acceptance, or treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: Accepting now would accept a correctness-only delivery: at head grader_digest() is 34928e69… while the published 300 v2 samples carry 0b75b5bb… (the body now labels them historical), no real two-arm skill experiment or performance ranking exists, and the full current-head source review the "Next action" asks for is still outstanding. A rejection discards the delivery.
Fix: Dong1017 records one comment accepting, revising or rejecting the CPU subset, the task/oracle interface and the delivery criteria — including whether historical samples plus the stated code-identity equivalence satisfy "retaining reproducible measurements" — after the full fd0f396 review; keep feat: add shared engineering task evaluation and Ascend RMSNorm #338 draft until then.

F3 — medium, human decision required — "Ownership and authorization" and the closing "Next step" still contradict the status block

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — unchanged "Ownership and authorization" section and closing "Next step" versus the new "Current state"/"Next action"; AGENTS.md:117.
Problem: The block records an assignee coordinating the scope decision and full current-head review plus a delivered implementation, while the section below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The new sentence "the original tracker authorization below is unchanged" now explicitly preserves that contradiction instead of marking it superseded.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117, or deciding whether benchmark work may proceed or be scheduled.
Impact: docs/agent-collaboration.md:14-15 requires recorded or linked authority; a record that answers both "assignee delivered and coordinating" and "no implementer, unauthorized, unscheduled" cannot serve as that authority, which is the ambiguity that let the pre-built delivery arise.
Fix: Rewrite the section and the closing step to the recorded state (accountable coordinator @Bounty-hunter, acceptance owner Dong1017, scope decision pending, initial subset and reuse choices not agreed), delete or mark the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half (subset and reuse choices agreed).

F4 — medium, human decision required — The four slices are still not children of the tracker, and #334–#336 still point at superseded snapshots

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []); superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336 bodies (links to ac03ae23/d0ce9d7); task: connect engineering tasks to existing Agent execution and evaluation #337 labels (only kind:task, area:workbench, last updated 2026-11-04T00:25:47Z); PR feat: add shared engineering task evaluation and Ascend RMSNorm #338.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links. task: connect engineering tasks to existing Agent execution and evaluation #337 was refreshed to fd0f396 but still lacks status:proposed/priority:P2/bot-reviewed, so it does not appear in the same decision queue as its siblings, while superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336 still link the older ac03ae23/d0ce9d7 revisions that the tracker body has replaced. PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 uses non-closing references, so merge progress rolls up nowhere.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records (tracker, four children, one draft PR) must be reconciled by hand; a decision owner opening a slice sees a superseded revision, and task: connect engineering tasks to existing Agent execution and evaluation #337 — which holds the supplemental results — is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues, align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels and refresh superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336's revision links to the tracker's pin; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record which minimal shared CPU engineering-benchmark subset, task/oracle contract and delivery criteria are agreed, and this edit adds the current-head review state of draft #338 (one open P2 canonical-task shadowing blocker) plus the 49-test software-control evidence. Verdict: human decision required — the edit's checkable claims hold, including the reported blocker, but no accept/revise/reject decision is recorded and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). F2, F5, F6 remain fixed; F7 remains answered (the 300 v2 samples are labelled historical); F8 is fixed by the new "improve integration" wording; F1, F3 and F4 remain open. No label value is suggested: every axis already carries a value.

Verified this round: PR #338 head equals the pinned fd0f396c (9 commits, draft), review 5761812435 is CHANGES_REQUESTED at that head with exactly one inline comment (r4462417808, src/inferrsi/harness/skill_eval_benchmark.py:50), no later review, reply or commit exists, the dfcc1855→fd0f396c delta is the three documentation/evidence files named, run 41746656869 is green/skipped per its seven jobs at that head, and the shadowing defect is real in source: contract.load_task resolves an existing Path(name) directory before the packaged task (src/inferrsi/harness/benchmarks/contract.py:153-158 at the head), while task_descriptor/verify gate on builtin_tasks() and then call load_task (src/inferrsi/harness/skill_eval_benchmark.py:48-50, :271), so a same-named cwd package is frozen into the descriptor and revalidated against itself. No new factual defect found in this edit.

F1 — high, human decision required — The edit records delivery plus an open changes-requested review, but still no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 review 5761812435 and comment r4462417808; AGENTS.md:117; docs/agent-collaboration.md:13-15.
Problem: Checked this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no comment newer than the 2026-11-04T00:38Z bot review and one body edit; PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 has exactly four reviews, the newest being the author-independent CHANGES_REQUESTED review at head, which asks for a packaged-resource loader and a cwd-shadow regression; superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 carry no acceptance comment. The body itself concedes contributor-reported delegation "does not replace maintainer scope acceptance". Assignment and a review record identify accountability, not authority (docs/agent-collaboration.md:14-15).
Conditions: Any move to merge or advance feat: add shared engineering task evaluation and Ascend RMSNorm #338, mark child acceptance, or treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: The decision owner would be accepting a delivery that currently has an open correctness blocker: at head the formal adapter can resolve a cwd-shadow package whose oracle tolerance is then frozen as the task identity, so "without duplicating task truth" (acceptance item 4) is unmet until that review is satisfied. A rejection discards a nine-commit delivery.
Fix: Dong1017 records one comment accepting, revising or rejecting the CPU subset, the task/oracle interface and the delivery criteria — stating whether historical 300-sample evidence plus the code-identity argument satisfies "retaining reproducible measurements" and whether acceptance waits for the shadowing fix — and feat: add shared engineering task evaluation and Ascend RMSNorm #338 stays draft until then.

F3 — medium, human decision required — "Ownership and authorization" and the closing "Next step" still contradict the status block

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — unchanged "Ownership and authorization" section and closing "Next step" versus the new "Current state"/"Next action"; AGENTS.md:117.
Problem: The block now records an assignee coordinating the scope decision and the current-head re-review of a delivered implementation, while the section below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The new sentence "the original tracker authorization below is unchanged" now explicitly preserves that contradiction. The new "Next action" is at least tracking-phrased ("retains the reported canonical-task correction; track its disposition and current-head re-review"), which is an improvement over the previous work instruction.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 (confirm the governing Issue's accepted scope before changing code), or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record answers both "assignee coordinating delivery and re-review" and "no implementer, implementation unauthorized, not yet scheduled", which is the ambiguity that allowed the pre-built delivery to arise.
Fix: Rewrite the section and closing step to the recorded state (accountable coordinator @Bounty-hunter, acceptance owner Dong1017, scope decision pending, initial subset and reuse choices not agreed), mark the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half.

F4 — medium, human decision required — The four slices are still not children of the tracker, and #334–#336 still point at superseded snapshots

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []); superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336 bodies (links to ac03ae23/d0ce9d7); task: connect engineering tasks to existing Agent execution and evaluation #337 labels (only kind:task, area:workbench, updated 2026-11-04T00:25:47Z); PR feat: add shared engineering task evaluation and Ascend RMSNorm #338.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links. task: connect engineering tasks to existing Agent execution and evaluation #337 links fd0f396c but still lacks status:proposed/priority:P2/bot-reviewed, so it is outside the same decision queue as superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336, while superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336 still link the older revisions the tracker body has replaced. PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 uses non-closing references, so merge progress rolls up nowhere and the new P2 blocker has no surface on the child that would track the fix.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records (tracker, four children, one draft PR) must be reconciled by hand; a decision owner opening a slice sees a superseded revision, and task: connect engineering tasks to existing Agent execution and evaluation #337 is easy to miss.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues, align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels, and refresh superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336's revision links to the tracker's pin; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

F1 — high, human decision required — The edit records delivery plus an open changes-requested review, but still no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 review 5761812435 and comment r4462417808; AGENTS.md:117; docs/agent-collaboration.md:13-15.
Problem: At this scan [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no comment newer than the 2026-11-04T00:38Z bot review and one body edit; PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 has exactly four reviews, the newest being the CHANGES_REQUESTED review at head, which asks for a packaged-resource loader and a cwd-shadow regression; superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 carry no acceptance comment. The body itself concedes contributor-reported delegation "does not replace maintainer scope acceptance". Assignment and a review record identify accountability, not authority (docs/agent-collaboration.md:14-15).
Conditions: Any move to merge or advance feat: add shared engineering task evaluation and Ascend RMSNorm #338, mark child acceptance, or treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: The decision owner would be accepting a delivery that currently has an open correctness blocker: at head the formal adapter can resolve a cwd-shadow package whose oracle tolerance is then frozen as the task identity, so "without duplicating task truth" (acceptance item 4) is unmet until that review is satisfied. A rejection discards a nine-commit delivery.
Fix: Dong1017 records one comment accepting, revising or rejecting the CPU subset, the task/oracle interface and the delivery criteria — stating whether historical 300-sample evidence plus the code-identity argument satisfies "retaining reproducible measurements" and whether acceptance waits for the shadowing fix — and feat: add shared engineering task evaluation and Ascend RMSNorm #338 stays draft until then.

Status: open (delivery and its remaining blocker are recorded; the decision is not).

F3 — medium, human decision required — "Ownership and authorization" and the closing "Next step" still contradict the status block

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — unchanged "Ownership and authorization" section and closing "Next step" versus the new "Current state"/"Next action"; AGENTS.md:117.
Problem: The block now records an assignee coordinating the scope decision and the current-head re-review of a delivered implementation, while the section below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The new sentence "the original tracker authorization below is unchanged" now explicitly preserves that contradiction. The new "Next action" is at least tracking-phrased ("retains the reported canonical-task correction; track its disposition and current-head re-review"), which is an improvement over the previous work instruction.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 (confirm the governing Issue's accepted scope before changing code), or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record answers both "assignee coordinating delivery and re-review" and "no implementer, implementation unauthorized, not yet scheduled", which is the ambiguity that allowed the pre-built delivery to arise.
Fix: Rewrite the section and closing step to the recorded state (accountable coordinator @Bounty-hunter, acceptance owner Dong1017, scope decision pending, initial subset and reuse choices not agreed), mark the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half.

Status: open.

F4 — medium, human decision required — The four slices are still not children of the tracker, and #334–#336 still point at superseded snapshots

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []); superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336 bodies (links to ac03ae23/d0ce9d7); task: connect engineering tasks to existing Agent execution and evaluation #337 labels (only kind:task, area:workbench, updated 2026-11-04T00:25:47Z); PR feat: add shared engineering task evaluation and Ascend RMSNorm #338.
Problem: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links. task: connect engineering tasks to existing Agent execution and evaluation #337 links fd0f396c but still lacks status:proposed/priority:P2/bot-reviewed, so it is outside the same decision queue as superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336, while superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336 still link the older revisions the tracker body has replaced. PR feat: add shared engineering task evaluation and Ascend RMSNorm #338 uses non-closing references, so merge progress rolls up nowhere and the new P2 blocker has no surface on the child that would track the fix.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records (tracker, four children, one draft PR) must be reconciled by hand; a decision owner opening a slice sees a superseded revision, and task: connect engineering tasks to existing Agent execution and evaluation #337 is easy to miss.
Fix: The maintainer decides whether to adopt the split; if yes, add superseded: benchmark research moved to task library and integration #334–task: connect engineering tasks to existing Agent execution and evaluation #337 as native sub-issues, align task: connect engineering tasks to existing Agent execution and evaluation #337's status/priority labels, and refresh superseded: benchmark research moved to task library and integration #334–tracking: grow reusable engineering optimization tasks #336's revision links to the tracker's pin; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Status: open (verified unchanged this round).

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

Navigation Menu

Repository navigation

[Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252

Goal

Candidate coverage, to narrow before implementation

Minimal shared contract to discuss

Relationship to existing work

Tracking and acceptance

Ownership and authorization

[RFC] Optimization search policy: candidate admission, cost ladder and stop contract

1. Problem

2. Design principles

3. Objects

3.1 Search-space cell index

3.2 Candidate card

3.3 Admission gates

3.4 Cost ladder → `RunSpec`

3.5 Phase-locked round

3.6 Stop evaluator

3.7 Negative-result ledger

4. Relationship to existing work

5. Dependency and ordering

6. Delivery slices

7. Acceptance criteria for this RFC

8. What this RFC does not authorize

9. Open questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252

Description

Goal

Candidate coverage, to narrow before implementation

Minimal shared contract to discuss

Relationship to existing work

Tracking and acceptance

Ownership and authorization

Activity

amy-why-3459 commented on Oct 29, 2026

[RFC] Optimization search policy: candidate admission, cost ladder and stop contract

1. Problem

2. Design principles

3. Objects

3.1 Search-space cell index

3.2 Candidate card

3.3 Admission gates

3.4 Cost ladder → RunSpec

3.5 Phase-locked round

3.6 Stop evaluator

3.7 Negative-result ledger

4. Relationship to existing work

5. Dependency and ordering

6. Delivery slices

7. Acceptance criteria for this RFC

8. What this RFC does not authorize

9. Open questions

Dong1017 commented on Oct 29, 2026

Bounty-hunter commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 4, 2026

agent-ci-infra-bot commented on Nov 4, 2026

agent-ci-infra-bot commented on Nov 4, 2026

agent-ci-infra-bot commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

3.4 Cost ladder → `RunSpec`