Repository navigation
Open
Description
Background
The strategic roadmap defines 1.0.0 Beta release criteria that are broader than unit-test/CI success:
- install, upgrade, run, dashboard, benchmark, and migration flows work on clean projects;
- at least three representative project types have documented smoke results;
- failure learning produces reviewed rule candidates without silently hardening bad rules;
- bilingual docs explain the core workflow without maintainer context;
- public claims are tied to WORKFLOW_EVAL, benchmark output, or release evidence.
The repository currently reports package version 0.58.0. Automated suites and individual demo assets are useful, but they do not by themselves constitute a reproducible, documented three-project release qualification run.
Goal
Create an evidence-backed 1.0.0 Beta qualification pack. Do not claim release readiness from unit tests alone.
Required work
- Define three representative clean-project fixtures (for example: Node/TypeScript library, Python service, and a multi-package or full-stack repository). Confirm the final set from actual support and CI capability.
- For each fixture, run and retain machine-readable evidence for:
- install and project initialization;
- upgrade/check and migration compatibility;
- AI OS plan + governed run;
- dashboard health and Agent Control readiness;
- benchmark and WORKFLOW_EVAL;
- smoke/preflight/verification and package/build checks.
- Record exact commit, package version, OS/runtime, commands, exit codes, report paths, and residual risks. Separate CI fixture results from real clean-project smoke runs.
- Prove failure-learning candidates remain review-only: no silent promotion into production rules; include accept/reject/replay evidence.
- Audit Chinese/English parity for install, upgrade, AI OS, dashboard, smoke, and failure recovery docs.
- Reconcile strategic positioning and README claims with measured evidence. No guaranteed token, hallucination, or productivity percentages without benchmark data.
Acceptance criteria
- Three clean representative projects have reproducible smoke reports checked into a curated evidence location.
- Install, upgrade, migration, run, dashboard, benchmark, and verification paths pass for the supported project matrix, or have explicit scoped blockers.
- Failure learning is shown to produce reviewable candidates without silently changing production policy.
- Chinese and English quickstart/troubleshooting guidance is materially equivalent.
- Public claims are traceable to benchmark/eval/release artifacts.
- A release reviewer can reproduce the qualification from documented commands without maintainer-only context.
Non-goals
- Do not publish v1.0.0 as part of this issue.
- Do not invent benchmark results or check in machine-local secrets/runtime state.
- Do not broaden supported platforms/project types without evidence.
Initial audit
- Current package version: 0.58.0.
- docs/AI_ENGINEERING_OS_POSITIONING.md already defines the three-project smoke and evidence-backed public-claims criteria.
- Existing assets include docs/WORKFLOW_EVAL.md, docs/start/customer-smoke.md, docs/start/agent-governance-demo.md, and examples/demo-projects/agent-governance-demo; these are starting points, not proof of three completed project qualifications.
- Recent P0 issues [1.0][P0] 明确 AI OS governed run 与真实 Agent execution 语义 #47 and [1.0][P0] 建立 Shield/Guardrails 跨平台回归样本库 #48 are closed; their work improves semantic and regression safety but does not replace end-to-end beta qualification.
Metadata
Metadata
Assignees
Labels
No labels
Activity
hongmaple0820 commentedon Nov 16, 2026
推进记录:新增 Draft PR #79(https://github.com/hongmaple0820/scale-engine/pull/79),提交三类 clean-project beta qualification runbook 与机器可读证据记录模板,覆盖 Node/TypeScript、Python service、multi-package/full-stack,另含 failure-learning review-only 验证、双语文档 parity 与 public-claim evidence gate。明确说明这只是可复现执行规程,不是三项目已通过的证据;三个真实 clean-project runs 仍未执行,Issue #71 保持 open。
hongmaple0820 commentedon Nov 17, 2026
Qualification progress (2026-11-17): PR #79's runbook has now passed Source CI, Coverage, and Published Package Gate on its current head. I added stacked Draft PR #82 (#82), which introduces a machine-checkable evidence-record validator: a record cannot claim
passedunless all eight required stages are present and passed, source/fixture commits are full SHAs, exit codes agree with stage status, and a reviewer has approved it. This is evidence-integrity tooling only; it does not execute or claim completion of the three clean-project runs. Those runs remain outstanding.hongmaple0820 commentedon Nov 17, 2026
Qualification tooling follow-up (2026-11-17): PR #82 now also validates timezone-qualified stage timestamps and their ordering, requires stdout/stderr evidence metadata, and rejects absolute or parent-traversal artifact paths. During review I found two regression-test cases were building incomplete stage fixtures (missing stdout/stderr fields); fixed the shared fixture helper in commit
0b8974734554c91957c4ad84dd04fdee21abcc9aso those tests exercise the intended artifact-path checks. On this head, Coverage and Published Package Gate (Ubuntu/macOS) are green; Source CI is still running. This remains validator/test hardening only. The three actual clean-project qualifications and failure-learning accept/reject/replay evidence are still outstanding; no Beta readiness claim is made.hongmaple0820 commentedon Nov 17, 2026
PR #82 follow-up (latest head
eea2703463a70a8456b800a1cf74041ea72511c6): while auditing the validator against its stated relative-path contract, I foundstderrArtifactaccepted any non-empty string, including absolute machine-local paths. Tightened it to use the same relative-path / parent-traversal check as stdout and report artifacts, and added a dedicated regression for an absolute Windows stderr path. New Source CI, Coverage, and Published Package Gate runs have started for this head; results are pending. The earlier head's broad CI matrix had all jobs green except the final macOS Node 22 test still running. This is validator hardening only: no clean-project qualification has yet been executed, so Beta readiness remains unproven.hongmaple0820 commentedon Nov 17, 2026
Latest PR #82 verification update: head
eea2703463a70a8456b800a1cf74041ea72511c6now has all three required CI workflows green — Source CI, Coverage, and Published Package Gate. The prior absolutestderrArtifactpath gap is covered by the new regression. I also audited the failure-learning path against Issue #71:scale eval run -> failures -> replay -> promote-failureis covered by CLI tests, and promotion creates animprovementscandidate rather than editing policy files. However, there is no persisted CLI accept/reject/review lifecycle for improvement candidates;RuleMaturity.approveRuleMaturityis a library-level in-memory gate, not an end-to-end review/replay workflow. I am recording this as a remaining acceptance gap rather than treating candidate creation as proof of reviewed learning. Three clean-project qualification runs are still outstanding.hongmaple0820 commentedon Nov 17, 2026
Failure Learning review lifecycle implementation is now in draft PR #83: #83
This adds persistent one-time candidate accept/reject decisions with reviewer identity, rationale, and timestamp; CLI listing/review commands; and tests for missing reviewer, duplicate decisions, and production-policy immutability. Accepted candidates are only marked
shadowStatus: pending; no replay is claimed or executed, and no production policy/hook is modified. Executable Shadow replay and persistent RuleMaturity integration remain open follow-up work. CI is still running; this is not yet validated or merged. The three clean-project qualification runs remain outstanding.hongmaple0820 commentedon Nov 18, 2026
Follow-up implementation is in stacked Draft PR #84: #84 (base: #83).
It adds a persistent RuleMaturity evidence ledger under
.scale/evals/rule-maturity/, allows only accepted improvement candidates to initialize non-blockingshadowrecords with rollback instructions, requires unique evidence IDs for recorded observations, and evaluates the existing maturity thresholds without changing the persisted stage. Store and CLI tests cover persistence, duplicate evidence rejection, threshold evaluation, and remaining inshadow.Important boundary: this is not yet an executable Shadow Replay engine.
shadow-hitrecords externally observed evidence; it does not run the proposed rule against live or replayed traffic. A passing evaluation only reportscandidate-hookas the next review stage. No production policy or hook is changed. CI is running; three clean-project qualification runs remain outstanding.hongmaple0820 commentedon Nov 18, 2026
CI follow-up for stacked PR #84: the first run failed immediately on a missing
from 'node:fs'clause inRuleMaturityStore.ts(TypeScript TS1005). This was corrected in commit90e8375cd6d06c48a7fa92e0030488bc79448119; a fresh Source CI, Coverage, and Package Gate run is now underway. The prior failed run is not being counted as a pass. PR #84 remains Draft and stacked on #83.hongmaple0820 commentedon Nov 18, 2026
PR #84 CI follow-up (head
d18558396787bdd6ae93589838b329958f7f7f83): the prior run exposed a malformed import intests/evolution/ruleMaturityStore.test.ts(frommissing); corrected in d185583. The same run's Source CI also failednpm audit --audit-level=high --omit=devon a high-severitypicomatchadvisory reachable from the production dependency graph. Coverage and the published-package gate passed on that prior run. Fresh Source CI and package-gate runs are underway; I am not marking PR #84 green until all required jobs complete, and the audit finding remains open for dependency-path/remediation review. This is GitHub CI evidence only, not one of the three clean-project qualification runs.hongmaple0820 commentedon Nov 18, 2026
Security-audit clarification for PR #84: the high-severity finding is GHSA-fc5q-39xc-vjpf (picomatch ReDoS). Current advisory data lists picomatch versions through 4.0.7 as affected and does not list a patched version. I will not suppress
npm auditor blindly force-upgrade transitive dependencies without evidence of remediation. Next step is to trace whether SCALE passes untrusted glob patterns into affected picomatch paths and document a defensible mitigation or wait for an upstream fixed release. The audit remains a blocker; no security exception is being claimed.hongmaple0820 commentedon Nov 18, 2026
Additional integrity hardening on PR #84:
shadow-initnow verifies that the accepted candidate's original Failure Replay record still exists and parses with the expected ID before creating a maturity record. Added a CLI regression test that removes the source failure artifact, expects initialization to fail closed, restores the fixture, and then verifies the normal path. Commit:687a5e899e1e5386f40834c78a87ba358ae27f34. This closes a concrete provenance gap; it does not implement replay execution.hongmaple0820 commentedon Nov 18, 2026
PR #84 updated its validation record with the exact Source CI audit evidence:
npm audit --audit-level=high --omit=devreports 9 production-tree advisories (7 high for picomatch GHSA-fc5q-39xc-vjpf; 2 moderate for Hono/@hono-node-server). The report traces picomatch throughchokidar -> anymatch/readdirpand@typescript-eslint/typescript-estree -> tinyglobby/fdir; npm's suggested--forceremediation would move chokidar 3 to 6 (breaking major), and the advisory lists no fixed picomatch version. I will not suppress this audit or force a major dependency migration without compatibility evidence. Also hardenedshadow-initto fail closed when its source Failure Replay artifact is missing/invalid; regression test added. Latest PR #84 head is now62c5c8b4a08351231fa069683721763c50573da4; fresh CI is pending.hongmaple0820 commentedon Nov 18, 2026
Shadow Replay design review finding: the existing
WorkflowEval.tsstoresreplayCommandfrom suite attempts and currently executes suite commands throughrunSafeCommand.SafeCommandRunnerdisables shell by default and rejects shell metacharacters, which is useful defense-in-depth, but it still launches the command's first token as an executable with caller-supplied arguments; it is not a sandbox, executable allowlist, network isolation, or filesystem write boundary. Therefore I will not wire storedreplayCommanddirectly into an automatic Shadow Replay runner. A safe runner needs a versioned replay spec, explicit executable/argument allowlist, isolated disposable worktree or fixture root, bounded timeout/output, controlled environment, network policy, and before/after artifact hashes; if those controls cannot be enforced portably, replay must remain manual with imported evidence. This is a deliberate safety gate, not a claim that replay exists.hongmaple0820 commentedon Nov 18, 2026
Further evidence integrity hardening in PR #84:
scale eval shadow-hitno longer accepts an arbitrary caller-supplied evidence ID. It now requires an existing regular file within the project directory, enforces a 10 MiB limit, computes SHA-256, and uses the digest as the evidence ID; same-content submissions are rejected even under different names. Path traversal/out-of-project paths fail closed. Added CLI regression coverage for path escape, digest identity and duplicate content; docs updated. Latest branch head after these changes isd02364063c75f58e5b7da0af6e62557f79f32434; fresh CI must pass before considering this change validated. Note this proves artifact existence/integrity, not that an actual Shadow Replay was executed.hongmaple0820 commentedon Nov 18, 2026
One more API-level integrity guard in PR #84:
PersistentRuleMaturityStore.recordHit()now rejects any evidence identifier that is notsha256:<64 lowercase hex>. This prevents callers bypassing the CLI's file hashing by submitting arbitrary labels directly to the store. Added a negative unit test and updated store fixtures to content-addressed IDs. Commit:4cf20098cb4bdca3a3293289b532c2b24aea7c0c. The CLI remains responsible for verifying the referenced file exists, is project-local, and is <=10 MiB before producing that digest.