Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
0f2916e
feat: add persistent shadow maturity evidence store
hongmaple0820 Nov 18, 2026
af4b7de
feat: add persistent shadow evidence CLI
hongmaple0820 Nov 18, 2026
60694b7
test: cover persistent shadow maturity evidence
hongmaple0820 Nov 18, 2026
a1aa58a
test: cover shadow evidence CLI lifecycle
hongmaple0820 Nov 18, 2026
11b4a62
docs: document persistent shadow evidence workflow
hongmaple0820 Nov 18, 2026
caaf62e
docs: document persistent shadow evidence workflow
hongmaple0820 Nov 18, 2026
fc6b77f
fix: serialize persistent shadow maturity updates
hongmaple0820 Nov 18, 2026
5265806
test: cover concurrent shadow maturity update lock
hongmaple0820 Nov 18, 2026
90e8375
fix: restore fs module import in shadow evidence store
hongmaple0820 Nov 18, 2026
d185583
fix: correct rule maturity test import syntax
hongmaple0820 Nov 18, 2026
ca75dd4
fix: require source failure evidence before shadow init
hongmaple0820 Nov 18, 2026
687a5e8
test: reject shadow init without source failure evidence
hongmaple0820 Nov 18, 2026
62c5c8b
test: normalize failure evidence assertion output
hongmaple0820 Nov 18, 2026
d9e513e
feat: require hashed in-project shadow evidence artifacts
hongmaple0820 Nov 18, 2026
0506816
test: require bounded in-project shadow evidence artifacts
hongmaple0820 Nov 18, 2026
52e5409
docs: require hashed project-local shadow evidence artifacts
hongmaple0820 Nov 18, 2026
d023640
docs: require hashed project-local shadow evidence artifacts
hongmaple0820 Nov 18, 2026
1b00c4a
fix: require content-addressed shadow evidence identifiers
hongmaple0820 Nov 18, 2026
4cf2009
test: enforce content-addressed maturity evidence ids
hongmaple0820 Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/EVOLUTION_SHADOW_MODE.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,14 @@ Promotion requires:
- `evaluateRulePromotion`
- `approveRuleMaturity`

## Persistent Shadow Evidence Ledger

Accepted workflow improvement candidates can be initialized as persistent, non-blocking maturity records with a documented rollback procedure. Records live under `.scale/evals/rule-maturity/`. Each observed hit requires an existing regular evidence file inside the project directory (maximum 10 MiB). The CLI records its SHA-256 digest as the evidence ID; duplicate content is rejected even if submitted under another filename. Paths escaping the project directory are rejected. False positives are counted against the configured promotion threshold.

The CLI exposes `shadow-init`, `shadow-hit`, `shadow-status`, and `shadow-evaluate`. These commands record external observations and calculate eligibility only. They do not execute the underlying rule against traffic, run a replay, change the maturity stage, generate hooks, or edit production policies. A passing evaluation reports `candidate-hook` as the next review stage while the persisted record remains `shadow`.

持久化 Shadow 证据台账支持为已接受的改进候选创建非阻断成熟度记录,并要求提供回滚方案。每次命中必须关联项目目录内已存在的普通证据文件(最大 10 MiB),CLI 以 SHA-256 生成证据 ID;重复内容和越界路径都会被拒绝。CLI 仅记录外部观察并计算门槛,不会自动执行规则或 Replay,不会改变成熟度阶段、生成 Hook 或修改生产策略。即使评估通过,也只报告下一审核阶段为 `candidate-hook`,记录本身仍保持 `shadow`。

## Hook Boundary

`HookGenerator` still requires `rule.approved === true`.
Expand Down
16 changes: 16 additions & 0 deletions docs/WORKFLOW_EVAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,22 @@ scale eval review-improvement <candidate-id> --decision reject --reviewer <ident

The review decision is immutable and requires both reviewer identity and rationale. Acceptance is review-only: it queues a pending Shadow validation marker, not an executed replay, a production rule change, or a blocking hook.

## Persistent Shadow Evidence / 持久化 Shadow 证据

After a candidate is explicitly accepted, initialize a non-blocking maturity record and append unique, externally observed evidence:

```bash
scale eval shadow-init <candidate-id> --rollback "Remove the shadow record; no hook is installed."
scale eval shadow-hit <candidate-id> --evidence-file shadow-evidence/<artifact-file>
scale eval shadow-hit <candidate-id> --evidence-file shadow-evidence/<artifact-file> --false-positive
scale eval shadow-status <candidate-id> --json
scale eval shadow-evaluate <candidate-id> --json
```

Records persist under `.scale/evals/rule-maturity/`. A shadow hit must reference an existing regular evidence file inside the project directory (maximum 10 MiB). The CLI records its SHA-256 digest as the evidence ID, so duplicate content is rejected even under another filename; paths escaping the project directory are rejected. Evaluation reports threshold eligibility only. Even when eligible, the stored stage remains `shadow`: this workflow does not execute a replay, promote a rule, generate a hook, or modify production policy. Evidence must come from a separately run and reviewable observation; do not use the command itself as proof that replay occurred.

候选必须先经人工接受,才能初始化非阻断成熟度记录。随后必须提交项目目录内已存在的普通证据文件(最大 10 MiB)。CLI 使用文件 SHA-256 作为证据 ID;即使更换文件名,内容相同也会被拒绝,不能重复累计命中数。越出项目目录的路径会被拒绝。记录保存在 `.scale/evals/rule-maturity/`。评估命令只报告是否满足晋级门槛;即使满足,持久化阶段仍保持 `shadow`。这些命令不会执行 Replay、晋级规则、生成 Hook 或修改生产策略。证据必须来自单独执行且可复核的观察,不能把记录命令本身当成 Replay 已运行的证明。

## Failure Replay To Memory

Failure Replay is local eval evidence first. When a failure pattern is useful for future work, ingest it into Memory Brain as an `incident` candidate:
Expand Down
141 changes: 139 additions & 2 deletions src/cli/evalCommands.ts
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
// SCALE Engine — Eval CLI Commands
// Extracted from api/cli.ts: workflow eval baseline, failure replay, and comparison commands.

import { createHash } from 'node:crypto'
import { defineCommand } from 'citty'
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'
import { dirname, resolve } from 'node:path'
import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from 'node:fs'
import { dirname, relative, resolve } from 'node:path'
import {
WorkflowEvalStore,
compareWorkflowEvalRuns,
renderWorkflowEvalReport,
runWorkflowEvalSuite,
} from '../eval/WorkflowEval.js'
import { createBenchmarkFromWorkflowEval, publishBenchmark } from '../eval/BenchmarkPublisher.js'
import { PersistentRuleMaturityStore } from '../evolution/RuleMaturityStore.js'
import { resolvePathWithinRoots } from '../core/pathSafety.js'
import { runGoldenSuite, type GoldenSuite } from '../eval/GoldenEval.js'
import { PROJECT_DIR, isTruthyFlag, resolveScaleDirForProject } from './engineBootstrap.js'
import { SCALE_ENGINE_VERSION } from '../version.js'
Expand Down Expand Up @@ -323,6 +326,136 @@ const evalReviewImprovement = defineCommand({
},
})

const evalShadowInit = defineCommand({
meta: { name: 'shadow-init', description: 'Create a persistent non-blocking Shadow maturity record from an accepted improvement' },
args: {
id: { type: 'positional', required: true, description: 'Accepted improvement candidate id' },
rollback: { type: 'string', required: true, description: 'Documented rollback procedure' },
dir: { type: 'string', default: PROJECT_DIR, description: 'Project directory' },
json: { type: 'boolean', default: false },
},
run({ args }) {
const projectDir = resolve(String(args.dir ?? PROJECT_DIR))
const scaleDir = resolveScaleDirForProject(projectDir)
const evalStore = new WorkflowEvalStore({ projectDir, scaleDir })
const candidate = evalStore.getImprovement(String(args.id))
if (!candidate) throw new Error(`Workflow improvement candidate not found: ${String(args.id)}`)
if (candidate.status !== 'accepted') {
throw new Error('Only an accepted improvement candidate can enter Shadow maturity tracking')
}
const sourceFailure = evalStore.getFailure(candidate.failureId)
if (!sourceFailure || sourceFailure.id !== candidate.failureId) {
throw new Error(`Source failure evidence is missing or invalid for improvement candidate: ${candidate.id}`)
}
const store = new PersistentRuleMaturityStore({ projectDir, scaleDir })
const record = store.create({
ruleId: candidate.id,
rollback: String(args.rollback),
defectEvidenceIds: [candidate.failureId],
})
if (args.json) {
console.log(JSON.stringify({ record, note: 'Evidence tracking only; no replay executed and no production policy or hook changed.' }, null, 2))
return
}
console.log(`Shadow maturity record created: ${record.ruleId}`)
console.log(' Stage: shadow (non-blocking)')
console.log(' Replay: not executed; submit observed evidence with shadow-hit')
console.log(' Production policies and hooks were not modified.')
},
})

const evalShadowHit = defineCommand({
meta: { name: 'shadow-hit', description: 'Record one externally observed Shadow evaluation with unique evidence' },
args: {
id: { type: 'positional', required: true, description: 'Shadow rule id (improvement candidate id)' },
'evidence-file': { type: 'string', required: true, description: 'Reviewable evidence artifact path within the project directory' },
'false-positive': { type: 'boolean', default: false, description: 'Mark this observation as a false positive' },
dir: { type: 'string', default: PROJECT_DIR, description: 'Project directory' },
json: { type: 'boolean', default: false },
},
run({ args }) {
const projectDir = resolve(String(args.dir ?? PROJECT_DIR))
const scaleDir = resolveScaleDirForProject(projectDir)
const evidencePath = resolvePathWithinRoots(String(args['evidence-file']), {
baseDir: projectDir,
allowedRoots: [projectDir],
label: 'Shadow evidence',
})
if (!existsSync(evidencePath) || !statSync(evidencePath).isFile()) {
throw new Error('Shadow evidence must be an existing regular file')
}
const evidenceStat = statSync(evidencePath)
if (evidenceStat.size > 10 * 1024 * 1024) {
throw new Error('Shadow evidence file exceeds the 10 MiB limit')
}
const digest = createHash('sha256').update(readFileSync(evidencePath)).digest('hex')
const evidenceId = `sha256:${digest}`
const store = new PersistentRuleMaturityStore({ projectDir, scaleDir })
const record = store.recordHit({
ruleId: String(args.id),
evidenceId,
falsePositive: Boolean(args['false-positive']),
})
const evidenceArtifact = { path: relative(projectDir, evidencePath), sha256: digest }
if (args.json) {
console.log(JSON.stringify({ record, evidenceArtifact }, null, 2))
return
}
console.log(`Shadow evidence recorded: ${record.ruleId}`)
console.log(` Artifact: ${evidenceArtifact.path}`)
console.log(` SHA-256: ${digest}`)
console.log(` Hits: ${record.shadowHits}`)
console.log(` False positives: ${record.falsePositiveCount}`)
},
})

const evalShadowStatus = defineCommand({
meta: { name: 'shadow-status', description: 'Show persistent Shadow maturity evidence' },
args: {
id: { type: 'positional', required: true, description: 'Shadow rule id' },
dir: { type: 'string', default: PROJECT_DIR, description: 'Project directory' },
json: { type: 'boolean', default: false },
},
run({ args }) {
const projectDir = resolve(String(args.dir ?? PROJECT_DIR))
const store = new PersistentRuleMaturityStore({ projectDir, scaleDir: resolveScaleDirForProject(projectDir) })
const record = store.get(String(args.id))
if (!record) throw new Error(`Shadow maturity record not found: ${String(args.id)}`)
if (args.json) {
console.log(JSON.stringify(record, null, 2))
return
}
console.log(`Shadow maturity: ${record.ruleId}`)
console.log(` Stage: ${record.stage}`)
console.log(` Hits: ${record.shadowHits}`)
console.log(` False positives: ${record.falsePositiveCount}`)
console.log(` Evidence ids: ${record.evidenceIds.join(', ') || '(none)'}`)
console.log(` Rollback: ${record.rollback}`)
},
})

const evalShadowEvaluate = defineCommand({
meta: { name: 'shadow-evaluate', description: 'Evaluate Shadow evidence against rule maturity thresholds without promoting a rule' },
args: {
id: { type: 'positional', required: true, description: 'Shadow rule id' },
dir: { type: 'string', default: PROJECT_DIR, description: 'Project directory' },
json: { type: 'boolean', default: false },
},
run({ args }) {
const projectDir = resolve(String(args.dir ?? PROJECT_DIR))
const store = new PersistentRuleMaturityStore({ projectDir, scaleDir: resolveScaleDirForProject(projectDir) })
const decision = store.evaluate(String(args.id))
if (args.json) {
console.log(JSON.stringify({ ...decision, note: 'Eligibility only; no stage promotion, hook generation, or production policy change.' }, null, 2))
return
}
console.log(`Shadow maturity eligible: ${decision.eligible ? 'yes' : 'no'}`)
console.log(` Next review stage: ${decision.nextStage}`)
for (const blocker of decision.blockers) console.log(` Blocker: ${blocker}`)
console.log(' No stage promotion or production policy change was performed.')
},
})

const evalGolden = defineCommand({
meta: { name: 'golden', description: 'P1-D: run the deterministic golden artifact-contract suite (CI merge gate)' },
args: {
Expand Down Expand Up @@ -366,6 +499,10 @@ export const evalCommand = defineCommand({
'promote-failure': evalPromoteFailure,
improvements: evalImprovements,
'review-improvement': evalReviewImprovement,
'shadow-init': evalShadowInit,
'shadow-hit': evalShadowHit,
'shadow-status': evalShadowStatus,
'shadow-evaluate': evalShadowEvaluate,
golden: evalGolden,
},
})
139 changes: 139 additions & 0 deletions src/evolution/RuleMaturityStore.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
import { closeSync, existsSync, mkdirSync, openSync, readFileSync, readdirSync, renameSync, unlinkSync, writeFileSync } from 'node:fs'
import { join, resolve } from 'node:path'
import {
createShadowRuleMaturity,
evaluateRulePromotion,
recordShadowHit,
type RuleMaturityRecord,
type RulePromotionDecision,
type RulePromotionThresholds,
} from './RuleMaturity.js'

export interface PersistentRuleMaturityStoreOptions {
projectDir?: string
scaleDir?: string
}

function safeId(value: string): string {
const id = value.trim()
if (!id || id === '.' || id === '..' || !/^[a-zA-Z0-9._-]+$/.test(id)) {
throw new Error('Invalid rule maturity id')
}
return id
}

function readRecord(path: string): RuleMaturityRecord | null {
if (!existsSync(path)) return null
try {
return JSON.parse(readFileSync(path, 'utf-8')) as RuleMaturityRecord
} catch {
throw new Error(`Invalid rule maturity record: ${path}`)
}
}

/**
* Persistent evidence ledger for non-blocking Shadow rules.
* It records observations and eligibility only; it never writes hooks or policies.
*/
export class PersistentRuleMaturityStore {
readonly projectDir: string
readonly recordsDir: string

constructor(options: PersistentRuleMaturityStoreOptions = {}) {
this.projectDir = resolve(options.projectDir ?? process.cwd())
const scaleDir = resolve(options.scaleDir ?? join(this.projectDir, '.scale'))
this.recordsDir = join(scaleDir, 'evals', 'rule-maturity')
}

private recordPath(ruleId: string): string {
return join(this.recordsDir, `${safeId(ruleId)}.json`)
}

get(ruleId: string): RuleMaturityRecord | null {
return readRecord(this.recordPath(ruleId))
}

list(): RuleMaturityRecord[] {
if (!existsSync(this.recordsDir)) return []
return readdirSync(this.recordsDir)
.filter(file => file.endsWith('.json'))
.map(file => readRecord(join(this.recordsDir, file)))
.filter((record): record is RuleMaturityRecord => Boolean(record))
.sort((a, b) => b.updatedAt - a.updatedAt)
}

create(input: { ruleId: string; rollback: string; defectEvidenceIds: string[] }): RuleMaturityRecord {
const ruleId = safeId(input.ruleId)
if (!input.rollback.trim()) throw new Error('A rollback method is required')
if (!input.defectEvidenceIds.length || input.defectEvidenceIds.some(id => !id.trim())) {
throw new Error('At least one defect evidence id is required')
}
return this.withLock(ruleId, () => {
mkdirSync(this.recordsDir, { recursive: true })
const path = this.recordPath(ruleId)
if (existsSync(path)) throw new Error(`Shadow maturity record already exists: ${ruleId}`)
const record = createShadowRuleMaturity({
ruleId,
rollback: input.rollback.trim(),
defectEvidenceIds: input.defectEvidenceIds,
})
this.write(record)
return record
})
}

recordHit(input: { ruleId: string; evidenceId: string; falsePositive?: boolean; now?: number }): RuleMaturityRecord {
return this.withLock(input.ruleId, () => {
const current = this.get(input.ruleId)
if (!current) throw new Error(`Shadow maturity record not found: ${input.ruleId}`)
const evidenceId = input.evidenceId.trim()
if (!/^sha256:[a-f0-9]{64}$/.test(evidenceId)) {
throw new Error('Shadow evidence id must be a SHA-256 digest in the form sha256:<64 lowercase hex characters>')
}
if (current.evidenceIds.includes(evidenceId)) {
throw new Error(`Shadow evidence already recorded: ${evidenceId}`)
}
const updated = recordShadowHit(current, {
evidenceId,
falsePositive: input.falsePositive,
now: input.now,
})
this.write(updated)
return updated
})
}

evaluate(ruleId: string, thresholds: RulePromotionThresholds = {}): RulePromotionDecision {
const record = this.get(ruleId)
if (!record) throw new Error(`Shadow maturity record not found: ${ruleId}`)
return evaluateRulePromotion(record, thresholds)
}

private withLock<T>(ruleId: string, operation: () => T): T {
mkdirSync(this.recordsDir, { recursive: true })
const lockPath = `${this.recordPath(ruleId)}.lock`
let lockFd: number
try {
lockFd = openSync(lockPath, 'wx')
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'EEXIST') {
throw new Error(`Shadow maturity record is being updated by another process: ${ruleId}`)
}
throw error
}
try {
return operation()
} finally {
closeSync(lockFd)
unlinkSync(lockPath)
}
}

private write(record: RuleMaturityRecord): void {
mkdirSync(this.recordsDir, { recursive: true })
const path = this.recordPath(record.ruleId)
const temporary = `${path}.${process.pid}.${record.updatedAt}.tmp`
writeFileSync(temporary, JSON.stringify(record, null, 2), { encoding: 'utf-8', flag: 'wx' })
renameSync(temporary, path)
}
}
Loading
Loading