Add deterministic compression signal engine#87
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
✅ Deploy Preview for comptext-v7 canceled.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the CompressionSignalEngine, a deterministic telemetry layer designed to monitor compression stability and classify cognitive modes such as 'habit' or 'deliberation'. The implementation includes core logic for signal calculation, hysteresis-based transitions, and a comprehensive test suite. Review feedback highlighted several critical improvements: refining the clamp01 function to correctly handle infinite spikes, resolving logic inconsistencies between stateless and stateful classification, optimizing redundant calculations in window building, and addressing an O(N^2) performance bottleneck in sequence evaluation by limiting history lookback.
| if (!Number.isFinite(value)) { | ||
| return 0; | ||
| } |
There was a problem hiding this comment.
The clamp01 function returns 0 for all non-finite numbers, including Infinity. In a signal engine, an infinite spike (e.g., in sparseFrameSpike) should logically result in a maximum error score of 1.0 rather than 0. Mapping Infinity to 0 could allow severe anomalies to bypass the trigger thresholds. Consider handling NaN specifically while allowing Infinity to be clamped to 1.
| if (!Number.isFinite(value)) { | |
| return 0; | |
| } | |
| if (Number.isNaN(value)) { | |
| return 0; | |
| } | |
| return Math.max(0, Math.min(1, value)); |
| export function classifyCognitiveMode(predictionError: number, thresholds: CompressionSignalThresholds = defaultCompressionSignalThresholds): CognitiveMode { | ||
| const score = clamp01(predictionError); | ||
| if (score >= thresholds.rollbackRequiredMin) { | ||
| return 'rollback_required'; | ||
| } | ||
| if (score < thresholds.habitMax) { | ||
| return 'habit'; | ||
| } | ||
| if (score <= thresholds.monitorMax) { | ||
| return 'monitor'; | ||
| } | ||
| if (score >= thresholds.deliberationMin) { | ||
| return 'deliberation'; | ||
| } | ||
| return 'monitor'; | ||
| } |
There was a problem hiding this comment.
The classifyCognitiveMode function implements a stateless classification that is inconsistent with the stateful logic used in the CompressionSignalEngine. It defines its own boundaries (like monitorMax vs deliberationMin) which creates a 'dead zone' or different behavior compared to the engine's hysteresis-based transitions. This duplication of logic is confusing and difficult to maintain.
| function buildSignalWindow(input: CompressionSignalInput, weights: CompressionSignalWeights): CompressionSignalWindow { | ||
| const tokenCostSpike = calculateTokenCostSpike(input.tokenEstimate, input.baselineTokenEstimate); | ||
| const replayConsistencyDrop = calculateReplayConsistencyDrop(input.replayConsistency, input.baselineReplayConsistency); | ||
| return Object.freeze({ | ||
| input: Object.freeze({ ...input, metadata: input.metadata === undefined ? undefined : Object.freeze({ ...input.metadata }) }), | ||
| compressionRatioDrop: calculateCompressionRatioDrop(input.compressionRatio, input.baselineCompressionRatio), | ||
| sparseFrameSpike: calculateSparseFrameSpike(input.sparseFrameRate, input.baselineSparseFrameRate), | ||
| unseenSignatureSignal: calculateUnseenSignatureSignal(input.unseenSignatureRate), | ||
| tokenCostSpike: input.tokenEstimate === undefined || input.baselineTokenEstimate === undefined ? undefined : tokenCostSpike, | ||
| replayConsistencyDrop: input.replayConsistency === undefined || input.baselineReplayConsistency === undefined ? undefined : replayConsistencyDrop, | ||
| predictionError: calculatePredictionError(input, weights), | ||
| }); | ||
| } |
There was a problem hiding this comment.
The buildSignalWindow function performs redundant calculations. It calls individual signal calculation functions (like calculateTokenCostSpike) and then calls calculatePredictionError, which repeats those same calculations internally. This can be optimized by calculating the signals once and using them for both the window object and the prediction error score.
function buildSignalWindow(input: CompressionSignalInput, weights: CompressionSignalWeights): CompressionSignalWindow {
const compressionRatioDrop = calculateCompressionRatioDrop(input.compressionRatio, input.baselineCompressionRatio);
const sparseFrameSpike = calculateSparseFrameSpike(input.sparseFrameRate, input.baselineSparseFrameRate);
const unseenSignatureSignal = calculateUnseenSignatureSignal(input.unseenSignatureRate);
const tokenCostSpike = calculateTokenCostSpike(input.tokenEstimate, input.baselineTokenEstimate);
const replayConsistencyDrop = calculateReplayConsistencyDrop(input.replayConsistency, input.baselineReplayConsistency);
const predictionError = roundSignal(
weights.compressionRatioDrop * compressionRatioDrop
+ weights.sparseFrameSpike * sparseFrameSpike
+ weights.unseenSignatureRate * unseenSignatureSignal
+ (weights.tokenCostSpike ?? 0) * tokenCostSpike
+ (weights.replayConsistencyDrop ?? 0) * replayConsistencyDrop,
);
return Object.freeze({
input: Object.freeze({ ...input, metadata: input.metadata === undefined ? undefined : Object.freeze({ ...input.metadata }) }),
compressionRatioDrop,
sparseFrameSpike,
unseenSignatureSignal,
tokenCostSpike: input.tokenEstimate === undefined || input.baselineTokenEstimate === undefined ? undefined : tokenCostSpike,
replayConsistencyDrop: input.replayConsistency === undefined || input.baselineReplayConsistency === undefined ? undefined : replayConsistencyDrop,
predictionError,
});
}| reasons.push('sparse_frame_rate increased'); | ||
| } | ||
| if (window.unseenSignatureSignal > 0) { | ||
| reasons.push(mode === 'deliberation' ? 'unseen_signature_rate remained elevated across debounce window' : 'unseen_signature_rate elevated'); |
There was a problem hiding this comment.
The reason string 'unseen_signature_rate remained elevated across debounce window' is potentially misleading. It is added whenever unseenSignatureSignal > 0 and the mode is 'deliberation', but being in deliberation mode doesn't guarantee that the unseen signature rate was the specific signal that triggered the transition or satisfied the debounce logic. It might be better to use a more neutral description unless specific attribution logic is implemented.
| const windows: CompressionSignalWindow[] = []; | ||
| let priorMode = initialMode; | ||
| return inputs.map((input) => { | ||
| const result = this.evaluateSignalWindow(input, windows, priorMode); |
There was a problem hiding this comment.
This implementation has O(N^2) time complexity because windows grows with each input and the entire array is passed to evaluateSignalWindow, which performs a spread operation [...priorWindows, window]. Since shouldEnterDeliberation only requires the last debounce.ofLast windows, you should only pass the necessary tail of the history to avoid performance degradation with large input sequences.
| const result = this.evaluateSignalWindow(input, windows, priorMode); | |
| const result = this.evaluateSignalWindow(input, windows.slice(-this.debounce.ofLast), priorMode); |
1 participant
Motivation
predictionErrorscore so downstream systems can gate deliberation or rollback.Description
dashboard/app/src/core/foundation/compressionSignals.tswith typed shapes (CognitiveMode,CompressionSignalInput,CompressionSignalWindow,CompressionSignalResult, weights, thresholds, hysteresis, and debounce).calculateCompressionRatioDrop,calculateSparseFrameSpike,calculateUnseenSignatureSignal, optional token/replay helpers), MVPcalculatePredictionError, and mode classification + sequence evaluation with hysteresis and debounce rules.dashboard/app/src/core/foundation/sampleData.tsand export the utilities viadashboard/app/src/core/foundation/index.ts.docs/compression_signal_engine.mddescribing purpose,Compression is perceptionclaim, input shape, formula, thresholds, hysteresis/debounce rules, cognitive modes, examples, limitations, and next integration points.tests/test_compression_signals_ts.pythat exercise calculations, classification, debounce/hysteresis, rollback immediate trigger, deterministic reason generation, and sample-data validity.Testing
pytest tests/test_compression_signals_ts.py -qand it passed (4 passed).npm run typecheckandnpm run buildfor the dashboard and showcase completed without errors.npm test/pytest -qreturned all tests green (existing suite plus new tests; overall56 passedshown during the run).npm run check(layout, typecheck, validate, build, test) completed with no failures.Codex Task