Skip to content

[1.0][P0] Prove AI Engineering OS beta readiness across representative projects #71

Open

Description

@hongmaple0820

Background

The strategic roadmap defines 1.0.0 Beta release criteria that are broader than unit-test/CI success:

  • install, upgrade, run, dashboard, benchmark, and migration flows work on clean projects;
  • at least three representative project types have documented smoke results;
  • failure learning produces reviewed rule candidates without silently hardening bad rules;
  • bilingual docs explain the core workflow without maintainer context;
  • public claims are tied to WORKFLOW_EVAL, benchmark output, or release evidence.

The repository currently reports package version 0.58.0. Automated suites and individual demo assets are useful, but they do not by themselves constitute a reproducible, documented three-project release qualification run.

Goal

Create an evidence-backed 1.0.0 Beta qualification pack. Do not claim release readiness from unit tests alone.

Required work

  1. Define three representative clean-project fixtures (for example: Node/TypeScript library, Python service, and a multi-package or full-stack repository). Confirm the final set from actual support and CI capability.
  2. For each fixture, run and retain machine-readable evidence for:
    • install and project initialization;
    • upgrade/check and migration compatibility;
    • AI OS plan + governed run;
    • dashboard health and Agent Control readiness;
    • benchmark and WORKFLOW_EVAL;
    • smoke/preflight/verification and package/build checks.
  3. Record exact commit, package version, OS/runtime, commands, exit codes, report paths, and residual risks. Separate CI fixture results from real clean-project smoke runs.
  4. Prove failure-learning candidates remain review-only: no silent promotion into production rules; include accept/reject/replay evidence.
  5. Audit Chinese/English parity for install, upgrade, AI OS, dashboard, smoke, and failure recovery docs.
  6. Reconcile strategic positioning and README claims with measured evidence. No guaranteed token, hallucination, or productivity percentages without benchmark data.

Acceptance criteria

  • Three clean representative projects have reproducible smoke reports checked into a curated evidence location.
  • Install, upgrade, migration, run, dashboard, benchmark, and verification paths pass for the supported project matrix, or have explicit scoped blockers.
  • Failure learning is shown to produce reviewable candidates without silently changing production policy.
  • Chinese and English quickstart/troubleshooting guidance is materially equivalent.
  • Public claims are traceable to benchmark/eval/release artifacts.
  • A release reviewer can reproduce the qualification from documented commands without maintainer-only context.

Non-goals

  • Do not publish v1.0.0 as part of this issue.
  • Do not invent benchmark results or check in machine-local secrets/runtime state.
  • Do not broaden supported platforms/project types without evidence.

Initial audit

Activity

  1. hongmaple0820 commented on Nov 16, 2026

    @hongmaple0820
    OwnerAuthor

    推进记录:新增 Draft PR #79(https://github.com/hongmaple0820/scale-engine/pull/79),提交三类 clean-project beta qualification runbook 与机器可读证据记录模板,覆盖 Node/TypeScript、Python service、multi-package/full-stack,另含 failure-learning review-only 验证、双语文档 parity 与 public-claim evidence gate。明确说明这只是可复现执行规程,不是三项目已通过的证据;三个真实 clean-project runs 仍未执行,Issue #71 保持 open。

  2. hongmaple0820 commented on Nov 17, 2026

    @hongmaple0820
    OwnerAuthor

    Qualification progress (2026-11-17): PR #79's runbook has now passed Source CI, Coverage, and Published Package Gate on its current head. I added stacked Draft PR #82 (#82), which introduces a machine-checkable evidence-record validator: a record cannot claim passed unless all eight required stages are present and passed, source/fixture commits are full SHAs, exit codes agree with stage status, and a reviewer has approved it. This is evidence-integrity tooling only; it does not execute or claim completion of the three clean-project runs. Those runs remain outstanding.

  3. hongmaple0820 commented on Nov 17, 2026

    @hongmaple0820
    OwnerAuthor

    Qualification tooling follow-up (2026-11-17): PR #82 now also validates timezone-qualified stage timestamps and their ordering, requires stdout/stderr evidence metadata, and rejects absolute or parent-traversal artifact paths. During review I found two regression-test cases were building incomplete stage fixtures (missing stdout/stderr fields); fixed the shared fixture helper in commit 0b8974734554c91957c4ad84dd04fdee21abcc9a so those tests exercise the intended artifact-path checks. On this head, Coverage and Published Package Gate (Ubuntu/macOS) are green; Source CI is still running. This remains validator/test hardening only. The three actual clean-project qualifications and failure-learning accept/reject/replay evidence are still outstanding; no Beta readiness claim is made.

  4. hongmaple0820 commented on Nov 17, 2026

    @hongmaple0820
    OwnerAuthor

    PR #82 follow-up (latest head eea2703463a70a8456b800a1cf74041ea72511c6): while auditing the validator against its stated relative-path contract, I found stderrArtifact accepted any non-empty string, including absolute machine-local paths. Tightened it to use the same relative-path / parent-traversal check as stdout and report artifacts, and added a dedicated regression for an absolute Windows stderr path. New Source CI, Coverage, and Published Package Gate runs have started for this head; results are pending. The earlier head's broad CI matrix had all jobs green except the final macOS Node 22 test still running. This is validator hardening only: no clean-project qualification has yet been executed, so Beta readiness remains unproven.

  5. hongmaple0820 commented on Nov 17, 2026

    @hongmaple0820
    OwnerAuthor

    Latest PR #82 verification update: head eea2703463a70a8456b800a1cf74041ea72511c6 now has all three required CI workflows green — Source CI, Coverage, and Published Package Gate. The prior absolute stderrArtifact path gap is covered by the new regression. I also audited the failure-learning path against Issue #71: scale eval run -> failures -> replay -> promote-failure is covered by CLI tests, and promotion creates an improvements candidate rather than editing policy files. However, there is no persisted CLI accept/reject/review lifecycle for improvement candidates; RuleMaturity.approveRuleMaturity is a library-level in-memory gate, not an end-to-end review/replay workflow. I am recording this as a remaining acceptance gap rather than treating candidate creation as proof of reviewed learning. Three clean-project qualification runs are still outstanding.

  6. hongmaple0820 commented on Nov 17, 2026

    @hongmaple0820
    OwnerAuthor

    Failure Learning review lifecycle implementation is now in draft PR #83: #83

    This adds persistent one-time candidate accept/reject decisions with reviewer identity, rationale, and timestamp; CLI listing/review commands; and tests for missing reviewer, duplicate decisions, and production-policy immutability. Accepted candidates are only marked shadowStatus: pending; no replay is claimed or executed, and no production policy/hook is modified. Executable Shadow replay and persistent RuleMaturity integration remain open follow-up work. CI is still running; this is not yet validated or merged. The three clean-project qualification runs remain outstanding.

  7. hongmaple0820 commented on Nov 18, 2026

    @hongmaple0820
    OwnerAuthor

    Follow-up implementation is in stacked Draft PR #84: #84 (base: #83).

    It adds a persistent RuleMaturity evidence ledger under .scale/evals/rule-maturity/, allows only accepted improvement candidates to initialize non-blocking shadow records with rollback instructions, requires unique evidence IDs for recorded observations, and evaluates the existing maturity thresholds without changing the persisted stage. Store and CLI tests cover persistence, duplicate evidence rejection, threshold evaluation, and remaining in shadow.

    Important boundary: this is not yet an executable Shadow Replay engine. shadow-hit records externally observed evidence; it does not run the proposed rule against live or replayed traffic. A passing evaluation only reports candidate-hook as the next review stage. No production policy or hook is changed. CI is running; three clean-project qualification runs remain outstanding.

  8. hongmaple0820 commented on Nov 18, 2026

    @hongmaple0820
    OwnerAuthor

    CI follow-up for stacked PR #84: the first run failed immediately on a missing from 'node:fs' clause in RuleMaturityStore.ts (TypeScript TS1005). This was corrected in commit 90e8375cd6d06c48a7fa92e0030488bc79448119; a fresh Source CI, Coverage, and Package Gate run is now underway. The prior failed run is not being counted as a pass. PR #84 remains Draft and stacked on #83.

  9. hongmaple0820 commented on Nov 18, 2026

    @hongmaple0820
    OwnerAuthor

    PR #84 CI follow-up (head d18558396787bdd6ae93589838b329958f7f7f83): the prior run exposed a malformed import in tests/evolution/ruleMaturityStore.test.ts (from missing); corrected in d185583. The same run's Source CI also failed npm audit --audit-level=high --omit=dev on a high-severity picomatch advisory reachable from the production dependency graph. Coverage and the published-package gate passed on that prior run. Fresh Source CI and package-gate runs are underway; I am not marking PR #84 green until all required jobs complete, and the audit finding remains open for dependency-path/remediation review. This is GitHub CI evidence only, not one of the three clean-project qualification runs.

  10. hongmaple0820 commented on Nov 18, 2026

    @hongmaple0820
    OwnerAuthor

    Security-audit clarification for PR #84: the high-severity finding is GHSA-fc5q-39xc-vjpf (picomatch ReDoS). Current advisory data lists picomatch versions through 4.0.7 as affected and does not list a patched version. I will not suppress npm audit or blindly force-upgrade transitive dependencies without evidence of remediation. Next step is to trace whether SCALE passes untrusted glob patterns into affected picomatch paths and document a defensible mitigation or wait for an upstream fixed release. The audit remains a blocker; no security exception is being claimed.

  11. hongmaple0820 commented on Nov 18, 2026

    @hongmaple0820
    OwnerAuthor

    Additional integrity hardening on PR #84: shadow-init now verifies that the accepted candidate's original Failure Replay record still exists and parses with the expected ID before creating a maturity record. Added a CLI regression test that removes the source failure artifact, expects initialization to fail closed, restores the fixture, and then verifies the normal path. Commit: 687a5e899e1e5386f40834c78a87ba358ae27f34. This closes a concrete provenance gap; it does not implement replay execution.

  12. hongmaple0820 commented on Nov 18, 2026

    @hongmaple0820
    OwnerAuthor

    PR #84 updated its validation record with the exact Source CI audit evidence: npm audit --audit-level=high --omit=dev reports 9 production-tree advisories (7 high for picomatch GHSA-fc5q-39xc-vjpf; 2 moderate for Hono/@hono-node-server). The report traces picomatch through chokidar -> anymatch/readdirp and @typescript-eslint/typescript-estree -> tinyglobby/fdir; npm's suggested --force remediation would move chokidar 3 to 6 (breaking major), and the advisory lists no fixed picomatch version. I will not suppress this audit or force a major dependency migration without compatibility evidence. Also hardened shadow-init to fail closed when its source Failure Replay artifact is missing/invalid; regression test added. Latest PR #84 head is now 62c5c8b4a08351231fa069683721763c50573da4; fresh CI is pending.

  13. hongmaple0820 commented on Nov 18, 2026

    @hongmaple0820
    OwnerAuthor

    Shadow Replay design review finding: the existing WorkflowEval.ts stores replayCommand from suite attempts and currently executes suite commands through runSafeCommand. SafeCommandRunner disables shell by default and rejects shell metacharacters, which is useful defense-in-depth, but it still launches the command's first token as an executable with caller-supplied arguments; it is not a sandbox, executable allowlist, network isolation, or filesystem write boundary. Therefore I will not wire stored replayCommand directly into an automatic Shadow Replay runner. A safe runner needs a versioned replay spec, explicit executable/argument allowlist, isolated disposable worktree or fixture root, bounded timeout/output, controlled environment, network policy, and before/after artifact hashes; if those controls cannot be enforced portably, replay must remain manual with imported evidence. This is a deliberate safety gate, not a claim that replay exists.

  14. hongmaple0820 commented on Nov 18, 2026

    @hongmaple0820
    OwnerAuthor

    Further evidence integrity hardening in PR #84: scale eval shadow-hit no longer accepts an arbitrary caller-supplied evidence ID. It now requires an existing regular file within the project directory, enforces a 10 MiB limit, computes SHA-256, and uses the digest as the evidence ID; same-content submissions are rejected even under different names. Path traversal/out-of-project paths fail closed. Added CLI regression coverage for path escape, digest identity and duplicate content; docs updated. Latest branch head after these changes is d02364063c75f58e5b7da0af6e62557f79f32434; fresh CI must pass before considering this change validated. Note this proves artifact existence/integrity, not that an actual Shadow Replay was executed.

  15. hongmaple0820 commented on Nov 18, 2026

    @hongmaple0820
    OwnerAuthor

    One more API-level integrity guard in PR #84: PersistentRuleMaturityStore.recordHit() now rejects any evidence identifier that is not sha256:<64 lowercase hex>. This prevents callers bypassing the CLI's file hashing by submitting arbitrary labels directly to the store. Added a negative unit test and updated store fixtures to content-addressed IDs. Commit: 4cf20098cb4bdca3a3293289b532c2b24aea7c0c. The CLI remains responsible for verifying the referenced file exists, is project-local, and is <=10 MiB before producing that digest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions