fix(bus): allow evaluate-experiment to override a stale stored baseline - #174
Open
asachs01 wants to merge 1 commit into
Open
fix(bus): allow evaluate-experiment to override a stale stored baseline#174asachs01 wants to merge 1 commit into
asachs01 wants to merge 1 commit into
Conversation
evaluate-experiment mechanically derives keep/discard by comparing the measured value against experiment.baseline_value, frozen at proposal time. That value can go stale by evaluation time — e.g. a baseline measured over a non-adjacent window that no longer represents a valid apples-to-apples comparison. Found live 2026-09-04 (task_1788524506203_29047861, marketing): exp_1786858829_uzaff mechanically read 'keep' comparing 48.65 against a stale 08-16 baseline of 37.6, while marketing's own fresh matched-window remeasurement (50.77 vs 48.65) showed a real decrease. The stored decision and the written learning text directly contradicted each other on the same completed record, with no CLI path to correct it after the fact. Adds an optional --baseline override to evaluate-experiment: - Requires --justification when used (same refuse-rather-than-guess discipline the function already applies to --score/measuredValue ambiguity) — an override needs to explain what changed and why the new value is valid. - experiment.baseline_value (the frozen historical fact of what was configured at proposal time) is never touched — only the local value used for this decision. - The override is recorded in the persisted learning text, not just an ephemeral console.error, so a reader of the completed record can see a decision was made against something other than the stored baseline without needing to cross-reference the justification separately. - On discard, next_baseline_value ratchets forward using the override (the corrected, valid comparison point) rather than the stale stored value, so the next cycle doesn't inherit the same staleness. 4 new tests reproduce the real bug (stale baseline mechanically reads keep with no override) and verify the fix (override flips the decision correctly, stored baseline_value stays frozen, learning text carries the override, and the keep/discard ratchet behaves correctly in both directions). The separate question of correcting an already-completed record whose decision was wrong before this fix existed is not addressed here — that remains the documented-correction-note workaround already established for this experiment family (same shape as exp_1787745238_vzgah), applied directly to exp_1786858829_uzaff's learnings.md as part of the same task.
|
Warning Review limit reachedNext included review available in 41 minutes. View limit detailsLimit details: You’ve used all 10 included reviews currently available. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (3)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evaluate-experimentmechanically derives keep/discard by comparing the measured value againstexperiment.baseline_value, frozen at proposal time. That value can go stale by evaluation time — e.g. a baseline measured over a non-adjacent window that no longer represents a valid apples-to-apples comparison.Found live 2026-09-04 (task_1788524506203_29047861, marketing):
exp_1786858829_uzaffmechanically readkeepcomparing 48.65 against a stale 08-16 baseline of 37.6, while marketing's own fresh matched-window remeasurement (50.77 vs 48.65) showed a real decrease. The storeddecisionand the writtenlearningtext directly contradicted each other on the same completed record, with no CLI path to correct it after the fact.What changed
Adds an optional
--baselineoverride toevaluate-experiment:--justificationwhen used (same refuse-rather-than-guess discipline the function already applies to--score/measuredValue ambiguity) — an override needs to explain what changed and why the new value is valid.experiment.baseline_value(the frozen historical fact of what was configured at proposal time) is never touched — only the local value used for this decision.learningtext, not just an ephemeralconsole.error, so a reader of the completed record can see a decision was made against something other than the stored baseline without cross-referencing the justification separately.next_baseline_valueratchets forward using the override (the corrected, valid comparison point) rather than the stale stored value, so the next cycle doesn't inherit the same staleness.The separate question of correcting an already-completed record whose decision was wrong before this fix existed is not addressed here — that remains the documented-correction-note workaround already established for this experiment family (same shape as
exp_1787745238_vzgah), applied directly toexp_1786858829_uzaff'slearnings.mdas part of the same task.Test plan
tests/sprint3-experiments.test.ts: reproduce the real bug (stale baseline mechanically reads keep with no override), verify the override flips the decision correctly, verify storedbaseline_valuestays frozen, verify the learning text carries the override, verify the keep/discard ratchet in both directionsnpx vitest run tests/sprint3-experiments.test.ts— 60/60 pass (56 pre-existing + 4 new), verified in an isolated worktreenpx tsc --noEmit— cleannpm run build— cleancortextos bus evaluate-experiment --help— new flag renders correctlyNote for reviewer: smoke-test footprint
CLI-level smoke-testing of the new flag (
create-experiment/run-experimentinvoked from a throwaway scratchagentDir) unexpectedly created two real, fleet-visible pending approvals —create-experiment's approval request lands in the shared approvals queue regardless of whichagentDirthe CLI was invoked from, even though the experiment JSON itself is correctly scoped to the scratch dir. Both were caught by boss and withdrawn (approval_1788524777_wy42i,approval_1788524782_dd9r3) same-day, no residue left in the real queue. Not a defect introduced by this PR — a pre-existing scoping gap between the two subsystems, noted here so it isn't mistaken for fallout from the change under review, and logged in analyst's MEMORY.md for future CLI smoke-testing.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.