Repository navigation
feat: add shared engineering task evaluation and Ascend RMSNorm - #338
Bounty-hunter wants to merge 19 commits into
Conversation
Add frozen task packages, independent CPU oracles and profiling, with bindings to the existing Workbench and offline SkillEval verifier. Include research decisions, reproducible integration checks and explicit qualification limits. Agent-assisted-by: Codex (implementation and documentation) Signed-off-by: dengyunyang <584797741@qq.com>
Dong1017
added
the
area:workbench
ChatGPT Codex Connector
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Scoped draft review of the new CPU benchmark measurement path against #252 and the campaign no-output-replay rule in #30. P2 benchmark-validity concern is attached inline: the precheck and timed samples reuse the same input object/content, so a memoizing candidate can appear faster without performing the operation on each call. Please address or explicitly constrain this before treating timing as optimization evidence. Static source review only; no tests, model runs, hardware measurements, or acceptance decision made here. The initial task subset remains subject to the recorded owner decision.
| if profiler: | ||
| profiler.enable() | ||
| started = time.perf_counter_ns() | ||
| output = compute(x, eps=oracle["eps"]) |
There was a problem hiding this comment.
P2 — measured calls can become cached-output replay. The precheck at line 94 already invokes compute(x), and this loop passes the same x object and values to every warmup/sample. A candidate with a process-local id(x)/content → output memo can pass every value/input-preservation assertion while most timed calls only return the stored output; the reported suite_latency_ms then measures replay, contrary to the campaign's no-final-output-caching boundary (#30). Suggest preparing a frozen, paired varying input sequence outside the timing boundary (same sequence for baseline and candidate) and validating each invocation; add a stateful memoizing-candidate negative regression. Public development workloads still cannot establish hostile-code resistance or held-out generalization.
There was a problem hiding this comment.
Addressed in ce84506e7106c4fc48fafdfab55d2ea2f9dffd50 with numpy-axis1-v2. Inputs/oracles are prepared before candidate invocation and retained for the complete action; each timed input has distinct bytes from precheck/warmup/other samples. Both arms and all consumers share the same deterministic per-call sequence. Each sample is bound to an input hash. The invariant zero case remains correctness-only, so the new score covers five varying workloads; old task/grader identities and evidence are preserved.
Regression controls confirm exact id/content memo implementations have no cache hits in the sequence, while shape/layout output replay is rejected without a metric. The fix does not claim hostile-code resistance. Forty focused tests and related CLI/SkillEval tests passed; actual Workbench and offline verifier execution retained 300 samples with matching planned and actual inputs. Statistical resolution remains unqualified.
Fix and validation · RSI consumer boundaries. Please re-review; this response does not mark the thread resolved or claim maintainer acceptance.
There was a problem hiding this comment.
The specific cached-output replay finding is addressed in ac03ae23; this is a scoped source confirmation, not approval of the whole draft.
- Inputs and independent oracles are prepared before candidate calls. All input objects stay alive, and later inputs must have distinct bytes; the zero case is correctness-only and five varying cases contribute timing.
- The new regressions require both identity-keyed and content-keyed memoization to miss, and reject stale shape/layout output replay without a usable performance metric. Per-sample input hashes and the frozen task/grader bindings are retained.
- @Bounty-hunter remains the implementer. Full-PR review, v2 timing/noise qualification and the recorded parent scope decision remain open; this confirmation establishes neither reliable performance ordering nor optimizer/skill benefit.
Evidence and limits
Assessed head: ac03ae23ce0bb5864a55c2c97d5f3d8d3b550d39; API base and actual main: 663f3b327998526a13aa31e72ad729025c18983c.
Read the current task contract, NumPy evaluator, worker, consumer/verifier bindings, both task manifests and the regression definitions, including input preparation and memo/replay controls. This is not a completed review of all 42 changed files.
The Python 3.11 hosted job reports Ruff success and 3,584 passed / 13 skipped / 284 subtests. It checked out merge 396688581967ea85d8edf2f7f169925546c5e207 (current head into the main above); the API comparison returned zero file differences from the standalone head. Draft qualification was skipped. I did not rerun tests or independently inspect the author's local 300-sample archive. No model/device execution or scope acceptance occurred.
Agent-assisted-by: Codex (validation fixture and documentation) Signed-off-by: dengyunyang <584797741@qq.com>
Document existing RSI consumers and remaining end-to-end acceptance. Agent-assisted-by: Codex (implementation, review and validation) Signed-off-by: dengyunyang <584797741@qq.com>
Agent-assisted-by: Codex (documentation) Signed-off-by: dengyunyang <584797741@qq.com>
Agent-assisted-by: Codex (review, implementation and documentation) Signed-off-by: dengyunyang <584797741@qq.com>
Agent-assisted-by: Codex (implementation, validation and documentation) Signed-off-by: dengyunyang <584797741@qq.com>
Agent-assisted-by: Codex (review) Signed-off-by: dengyunyang <584797741@qq.com>
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Provider/settings: Uncovered / unsupported paths (70)
|
Agent-assisted-by: Codex (validation) Signed-off-by: dengyunyang <584797741@qq.com>
Agent-assisted-by: Codex (validation and documentation) Signed-off-by: dengyunyang <584797741@qq.com>
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Incremental review at fd0f396c of the formal artifact integration since ac03ae23: one P2 correctness blocker, attached inline. The new built-in-only adapter can resolve a same-named working-directory task instead of the packaged task, and its freeze/revalidation path preserves that substitution.
The earlier cached-output replay finding remains addressed; I am not reopening or duplicating it. I inspected the new adapter, reused task/sandbox contracts, controller/run-trial/local-capture/report changes, regression definitions and CI wiring. Current Ready CI succeeds: the dedicated Linux controls report 49 passed with strict sandbox availability enabled; the 3.11 full check reports 3,631 passed, 18 skipped and 284 subtests. Actual checkout 98156741 has zero file differences from this head. Those tests do not cover cwd task shadowing.
Static source review plus existing hosted logs only; no local candidate/test execution, model or hardware work. Bounty-hunter retains the correction. Correctness-only limits, performance/skill-benefit gaps and #252's outstanding acceptance decision remain separate; this review does not accept the broader goal.
| def task_descriptor(name: str) -> dict: | ||
| if name not in builtin_tasks(): | ||
| raise ValueError("Only built-in shared CPU benchmark tasks are supported") | ||
| package = load_task(name) |
There was a problem hiding this comment.
P2 — resolve formal tasks from the packaged resource, not the working directory. Checking that name occurs in builtin_tasks() does not ensure that load_task(name) loads that built-in package: contract.load_task first prefers any existing Path(name) directory. validate_task(), starter() and verify() repeat the same lookup, so they all agree on a shadow rather than detect it.
For a concrete source-trace case, run from the InferRSI checkout with a local softmax-cpu/ copy of the packaged task and change only its oracle.atol to 1. Generate the descriptor through task_descriptor('softmax-cpu'). The manifest validator permits that positive finite tolerance. prepare() copies this same package, freezes its checkout-relative paths, and load_plan() compares against the same shadow, so the paired path also accepts it. A compute returning np.zeros_like(x) then satisfies all four output checks for every input: true softmax values lie in [0,1], so np.allclose(0, expected, atol=1) accepts them. The canonical packaged files need not change. This is static tracing/arithmetic, not a claim that I executed the candidate.
This makes a built-in-only, independently frozen correctness task depend on the caller's cwd and allows an incorrect artifact to pass under an accidentally or deliberately substituted rubric. It is separate from the documented public-suite/protocol-spoofing limits. Please use an explicit packaged-resource/built-in-only loader throughout the formal adapter and its prepare() dependency collection, while retaining path loading for the trusted Workbench API. Add a regression with a same-named cwd task directory that proves the canonical oracle/hash is used or the shadow is rejected.
There was a problem hiding this comment.
The specific cwd-shadowing finding is addressed at 6308268fc6d1fbf90e9ab97b5da8af79152f39ea (fix introduced in 00c41b24). This clears this finding only.
load_builtin_task()resolves the packaged resource and checks its manifest ID. The formal descriptor, starter and parent verifier now usetask_package(); prepare copies and freezes dependencies through that same canonical loader. Revalidation compares the retained package hash and starter against the canonical descriptor. Trusted Workbench path loading remains available.- The new regressions cover a same-named
softmax-cpu/cwd directory withatol=1, canonical descriptor/starter/oracle selection, prepare/dependency freezing and later revalidation. The real sandbox control rejects the zero-output implementation despite that shadow. The formal CPU path also rejects the new NPU adapter before execution. - I read the exact-head source and dedicated Linux control log: 54 passed, zero skips, with
INFERRSI_REQUIRE_BENCHMARK_SANDBOX=1and NumPy 2.2.6. The 3.11 full check reports 3,649 passed / 20 skipped / 284 subtests. CI checked out merge6f1506bc; its tree has zero file differences from this head. I did not execute code locally.
The separate NPU action/total-timeout finding remains open and is not superseded by this reply. This is not whole-PR approval, device qualification, performance/skill-benefit evidence, or scope acceptance; Bounty-hunter retains the implementation.
Agent-assisted-by: Codex (documentation and scope review) Signed-off-by: dengyunyang <584797741@qq.com>
Agent-assisted-by: Codex (source research and task design) Signed-off-by: dengyunyang <584797741@qq.com>
Bounty-hunter
changed the title
Keep reviewed path loading available to Workbench while resolving formal descriptors, starters, oracle checks and prepare dependencies only from packaged task resources. Reject unsupported device adapters before the NumPy sandbox. Agent-assisted-by: Codex (implementation and tests) Signed-off-by: dengyunyang <584797741@qq.com>
Define the affine device task and qualification flow; retain CPU results as historical software evidence and keep NPU SkillEval isolation separate. Agent-assisted-by: Codex (documentation) Signed-off-by: dengyunyang <584797741@qq.com>
Execute trusted candidates on an allocated NPU with an independent scalar oracle, synchronized eager timing, separate device profiling and Workbench A/A and reference controls. Agent-assisted-by: Codex (implementation and review) Signed-off-by: dengyunyang <584797741@qq.com>
A no-op comparison may fail either its speed gate or correctness. Accept the no-op control only after successful final quality validation, and compare its actual environment with every other result. Agent-assisted-by: Codex (implementation and review) Signed-off-by: dengyunyang <584797741@qq.com>
Match all action, trial and final validation identities against the pre-execution freeze, including incorrect controls stored outside the task package. Agent-assisted-by: Codex (implementation and review) Signed-off-by: dengyunyang <584797741@qq.com>
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Scoped incremental static review of the new rmsnorm-npu pilot in PR #338 (head 6308268f), against the execution limits recorded in Issue #337. One P2 execution-budget contract gap, attached inline: the 180-second action timeout is applied to standalone check/profile/negative controls but not to the A/A or reference Workbench trials. This can leave an individual trial running until the 8-minute comparison deadline; the script also does not itself enforce the separately stated 20-minute pilot envelope. Please bind the per-action and total deadlines or identify and verify the outer enforcement, with a timeout-control regression. This is a source/contract finding, not a hardware result. I inspected the new NPU adapter/driver, RunSpec, the Workbench runner/external/local executor chain and relevant tests. The earlier canonical-task shadowing review remains separate and is not repeated. No NPU/model/test execution or acceptance decision was performed.
| comparisons = {} | ||
| # Same implementation in both arms is the explicit no-op/A-A control. | ||
| for name, path in (("aa", baseline), ("reference", candidate)): | ||
| spec = task.runspec(path, repo=args.repo, output_dir=output / "results" / name, |
There was a problem hiding this comment.
P2 — the frozen per-action timeout is not applied to paired NPU trials. Issue #337's predeclared pilot says --timeout-seconds 180, two Workbench comparisons capped at 8 minutes each, and a 20-minute total runtime. Here args.timeout_seconds is passed only to run_action() for check/profile/negative controls (lines 96–116); task.runspec() does not carry it. run_experiment() passes its whole-comparison deadline to execute_trial() → execute_command(), which calls process.wait(timeout=deadline-now). Thus a first stalled A/A or reference trial may occupy almost the entire 480-second comparison instead of stopping at 180 seconds; two comparisons plus all standalone controls can also exceed 20 minutes unless an independently enforced outer deadline exists. That changes the frozen resource/lifecycle budget, particularly on a shared NPU allocation. Please cap each Workbench trial at min(comparison_remaining, action_timeout, pilot_remaining) (or implement equivalent tested enforcement), and include a deliberately nonterminating synthetic command in a CPU-only timeout regression. If a scheduler supplies the global bound, record its enforceable receipt and distinguish that from this script's guarantees.
Publish the NPU workflow, all timing samples and scoped keep decision; distinguish the trusted device pilot from formal artifact isolation and Agent efficacy. Agent-assisted-by: Codex (documentation and evidence review) Signed-off-by: dengyunyang <584797741@qq.com>
Bounty-hunter
changed the title
|
Review at head
|
Keep profiling optional, distinguish existing task tools from pending real-Agent integration, and preserve scoped evidence and unresolved review findings. Agent-assisted-by: Codex (documentation and scope review) Signed-off-by: dengyunyang <584797741@qq.com>
Bounty-hunter
changed the title
|
Incremental review This delta: the 12 files under Still open:
Static doc review of the delta only; I did not run the NPU tooling. |
Remove the CPU RMSNorm package and its dedicated historical probe. Keep Softmax regression coverage, preserve source-pinned experiment records, and retain the NPU task unchanged. Agent-assisted-by: Codex (implementation, test migration, and documentation) Signed-off-by: dengyunyang <584797741@qq.com>
|
Incremental review This delta looks consistent:
Still open:
Static review of the delta only; I did not run the tests or the NPU tooling. |
This PR supplies shared engineering task evaluation and the first Ascend NPU RMSNorm task. It extends the existing Workbench with task-owned correctness checks, synchronized measurements and optional profiling, and retains a real-device reference-control exercise.
Draft — foundation delivery, not completion of the real-Agent workflow. Current head:
9a755f97554d71a1a4b1d59200eddc30a11bdf01; NPU-tested source:6308268fc6d1fbf90e9ab97b5da8af79152f39ea. The latest change removes the built-in CPU RMSNorm task and its dedicated probe, migrates shared tests to Softmax, and updates current usage documentation. NPU task/worker and shared grader code are unchanged; device evidence retains its original source identity. The NPU per-action/total-timeout review remains unresolved; this PR is not ready to merge.Read first: 中文范围与两项工作 · existing capabilities and remaining integration · task-library plan · device evidence and limits.
Delivered behavior
check / measure / profile / compareoperations reuse the existing Workbench, AB/BA runner and reports.rmsnorm-npu/torch-npu-rmsnorm-v1acceptscompute(x, weight, *, eps)on contiguous 2D float32 NPU tensors. It checks against an independent scalar double/math.fsum oracle, validates shape/dtype/device and input/weight preservation, and never falls back to CPU.tools.npu_benchmark_validateexercises the packaged baseline/reference and incorrect controls, freezes identities, checks actual runtime consistency, and uses Workbench for A/A and reference AB/BA. It is explicitly launched and does not generate Agent candidates.softmax-cpuremains as the CPU software control.rmsnorm-cpuand its dedicated usefulness probe are removed; historical CPU experiment records remain source-pinned, with historical checkout links for reproduction. Shared correctness, replay-rejection, AB/BA and retained-artifact controls now use Softmax. The formal CPU artifact adapter remains. Formal task resolution uses canonical packaged resources; the formal adapter explicitly rejects NPU until its artifact execution boundary is qualified.Two work issues, two initial PRs
Former #334 research and #335 interface scopes are consolidated into #336/#337; their closure is an organizational disposition, not feature acceptance. Original research and interface documents remain.
The first delivery is this Draft PR. One subsequent Draft PR is planned, not yet created, for the bounded #337 integration and its verification; no separate report-only PR is planned. Later task additions belong to #336.
The intended workflow accepts built-in tasks or external projects with declared tests/benchmarks. Independent acceptance uses those fixed rules, not Agent-reported success. Callers control variables and repeat runs; the evaluator need not understand indexer or knowledge-library conditions. Profiling is optional. Existing #248 efficacy and #249/search-policy responsibilities remain separate; this PR does not add a general Agent platform or search controller.
Device evidence, with explicit limits
The plan was recorded before execution. One scheduled Ascend 910B1, CANN 9.1.0, torch 2.10.0+cpu with torch_npu 2.10.0.post4: baseline/reference each passed 375 assertions, six incorrect controls were rejected without metrics, and an additional replay control passed initial checks then failed on the first varying warmup. Actual traces show six composed compute kernels versus one RmsNorm per selected input.
[1,4096][64,4096]All 800 samples, controls and identities remain retained. Each comparison used four AB/BA pairs. The script's seven checks passed and the recorded runs exited successfully; this does not resolve the timeout-enforcement defect below. The original frozen plan, profiler checks and raw verdicts are unchanged by making profiling optional in the general workflow.
These observations concern an existing vendor fused reference under the recorded float32 workload. They do not demonstrate autonomous Agent discovery, statistically established general gains, model/service speedups, formal NPU isolation or RSI efficacy. Raw traces/logs remain in the private execution archive; the public extraction explains its omissions. Early failed probes and corrected test failures remain documented.
Validation and review status
softmax-cpuandrmsnorm-npu. The first Windows build hit a long-path error; the build succeeded with a shorter temporary extraction directory.Real NPU execution currently uses reviewed trusted source. #337 must establish the actual Agent-to-artifact handoff and a suitable NPU execution boundary; the CPU sandbox cannot qualify that by analogy. Runtime success is not project acceptance. PR remains Draft and GitHub reviewDecision remains CHANGES_REQUESTED.
Refs #252
Refs #336
Refs #337
Base:
663f3b327998526a13aa31e72ad729025c18983c. DCO and Agent-assisted-by attribution retained. Protected knowledge/policy paths are unchanged; no new model or device execution accompanies the CPU-task removal.