Skip to content

fix(bus): allow evaluate-experiment to override a stale stored baseline - #174

Open
asachs01 wants to merge 1 commit into
mainfrom
fix/experiment-baseline-override
Open

fix(bus): allow evaluate-experiment to override a stale stored baseline#174
asachs01 wants to merge 1 commit into
mainfrom
fix/experiment-baseline-override

Conversation

@asachs01

@asachs01 asachs01 commented Sep 4, 2026

Copy link
Copy Markdown

Summary

evaluate-experiment mechanically derives keep/discard by comparing the measured value against experiment.baseline_value, frozen at proposal time. That value can go stale by evaluation time — e.g. a baseline measured over a non-adjacent window that no longer represents a valid apples-to-apples comparison.

Found live 2026-09-04 (task_1788524506203_29047861, marketing): exp_1786858829_uzaff mechanically read keep comparing 48.65 against a stale 08-16 baseline of 37.6, while marketing's own fresh matched-window remeasurement (50.77 vs 48.65) showed a real decrease. The stored decision and the written learning text directly contradicted each other on the same completed record, with no CLI path to correct it after the fact.

What changed

Adds an optional --baseline override to evaluate-experiment:

  • Requires --justification when used (same refuse-rather-than-guess discipline the function already applies to --score/measuredValue ambiguity) — an override needs to explain what changed and why the new value is valid.
  • experiment.baseline_value (the frozen historical fact of what was configured at proposal time) is never touched — only the local value used for this decision.
  • The override is recorded in the persisted learning text, not just an ephemeral console.error, so a reader of the completed record can see a decision was made against something other than the stored baseline without cross-referencing the justification separately.
  • On discard, next_baseline_value ratchets forward using the override (the corrected, valid comparison point) rather than the stale stored value, so the next cycle doesn't inherit the same staleness.

The separate question of correcting an already-completed record whose decision was wrong before this fix existed is not addressed here — that remains the documented-correction-note workaround already established for this experiment family (same shape as exp_1787745238_vzgah), applied directly to exp_1786858829_uzaff's learnings.md as part of the same task.

Test plan

  • 4 new unit tests in tests/sprint3-experiments.test.ts: reproduce the real bug (stale baseline mechanically reads keep with no override), verify the override flips the decision correctly, verify stored baseline_value stays frozen, verify the learning text carries the override, verify the keep/discard ratchet in both directions
  • npx vitest run tests/sprint3-experiments.test.ts — 60/60 pass (56 pre-existing + 4 new), verified in an isolated worktree
  • npx tsc --noEmit — clean
  • npm run build — clean
  • cortextos bus evaluate-experiment --help — new flag renders correctly

Note for reviewer: smoke-test footprint

CLI-level smoke-testing of the new flag (create-experiment/run-experiment invoked from a throwaway scratch agentDir) unexpectedly created two real, fleet-visible pending approvals — create-experiment's approval request lands in the shared approvals queue regardless of which agentDir the CLI was invoked from, even though the experiment JSON itself is correctly scoped to the scratch dir. Both were caught by boss and withdrawn (approval_1788524777_wy42i, approval_1788524782_dd9r3) same-day, no residue left in the real queue. Not a defect introduced by this PR — a pre-existing scoping gap between the two subsystems, noted here so it isn't mistaken for fallout from the change under review, and logged in analyst's MEMORY.md for future CLI smoke-testing.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

evaluate-experiment mechanically derives keep/discard by comparing the
measured value against experiment.baseline_value, frozen at proposal
time. That value can go stale by evaluation time — e.g. a baseline
measured over a non-adjacent window that no longer represents a valid
apples-to-apples comparison.

Found live 2026-09-04 (task_1788524506203_29047861, marketing):
exp_1786858829_uzaff mechanically read 'keep' comparing 48.65 against a
stale 08-16 baseline of 37.6, while marketing's own fresh matched-window
remeasurement (50.77 vs 48.65) showed a real decrease. The stored
decision and the written learning text directly contradicted each other
on the same completed record, with no CLI path to correct it after the
fact.

Adds an optional --baseline override to evaluate-experiment:
- Requires --justification when used (same refuse-rather-than-guess
  discipline the function already applies to --score/measuredValue
  ambiguity) — an override needs to explain what changed and why the
  new value is valid.
- experiment.baseline_value (the frozen historical fact of what was
  configured at proposal time) is never touched — only the local value
  used for this decision.
- The override is recorded in the persisted learning text, not just an
  ephemeral console.error, so a reader of the completed record can see
  a decision was made against something other than the stored baseline
  without needing to cross-reference the justification separately.
- On discard, next_baseline_value ratchets forward using the override
  (the corrected, valid comparison point) rather than the stale stored
  value, so the next cycle doesn't inherit the same staleness.

4 new tests reproduce the real bug (stale baseline mechanically reads
keep with no override) and verify the fix (override flips the decision
correctly, stored baseline_value stays frozen, learning text carries
the override, and the keep/discard ratchet behaves correctly in both
directions).

The separate question of correcting an already-completed record whose
decision was wrong before this fix existed is not addressed here — that
remains the documented-correction-note workaround already established
for this experiment family (same shape as exp_1787745238_vzgah), applied
directly to exp_1786858829_uzaff's learnings.md as part of the same task.
@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 41 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: ce6a7223-d87e-4eb0-928f-2a756891388f

📥 Commits

Reviewing files that changed from the base of the PR and between e44bf20 and 11825ce.

📒 Files selected for processing (3)
  • src/bus/experiment.ts
  • src/cli/bus.ts
  • tests/sprint3-experiments.test.ts

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant