docs: append cross-session agent know-how on nim_benchmark date gates and review sidecar gap - #1076
Draft
seonghobae wants to merge 1 commit into
Draft
docs: append cross-session agent know-how on nim_benchmark date gates and review sidecar gap#1076seonghobae wants to merge 1 commit into
seonghobae wants to merge 1 commit into
Conversation
… and review sidecar gap Append-only additions to AGENTS.md (two new sections), a one-line pointer in CLAUDE.md, and a CHANGELOG.md entry, recording lessons independently re-verified against this exact checkout and live GitHub state on 2026-09-05: the nim_benchmark.py evidence-gate recurring bug class (hardcoded valid_until_date literals causing silent branch-coverage loss once lapsed, diagnosed/repaired three times this cycle across PR #1070, PR #1071, and issue #1075), and the still-open central review sidecar/egress gap tracked in issue #1041 comment 5550412102 and ContextualWisdomLab/.github issue #1759, including the PR #1884 conflation-then-correction cautionary tale. Documentation only: no code, test, or behavior change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documentation-only. Appends two new
AGENTS.mdsections (with a pointer bullet inCLAUDE.md):Recurring bug class: hardcoded review-cadence dates.
nim_benchmark.py's fail-closed evidence gates (ACTUAL_COST_EVIDENCE["valid_until_date"], and any pricing-scenario's ownvalid_until_date) are deliberately literal calendar dates meant to lapse and force a human re-review — never "fix" a lapsed date by rewriting the production literal, that defeats the gate. This exact bug class has now bitten three separate test files this cycle (tests/test_nim_benchmark.py,tests/test_spend_analytics.py,tests/test_nim_benchmark_release_acceptance.py—#1070/#1071/#1075). The established fix pattern is documented: an opt-in, not autouse,current_actual_cost_evidencefixture requested by name only from tests that need to get past the gate.Central review sidecar/egress gap (tracked, not yet closed).
scripts/ci/contextual_orchestrator_review_sidecar.sh(vendored by.github's four review-pipeline consumers) still injects all five raw provider secrets, clones and builds this repo fresh on the calling runner per invocation, and runs discovery in-process there. Citations:#1041comment5550412102(six requirements for the immutable release this needs),ContextualWisdomLab/.github#1759.See the companion
ContextualWisdomLab/.githubPR for the shared, org-wide half of this same effort (Codex-liveness, the GitHub Project-protocol MCP capability gap, and the@openai/codexCLI-invocation finding).Verification
python -m pytest tests/test_nim_benchmark.py tests/test_nim_benchmark_release_acceptance.py tests/test_conventions.py -qandpython -m interrogate -f 100 contextual_orchestrator/nim_benchmark.pyboth pass clean on this diff.git diffagainstmainis strictly additive toAGENTS.md/CLAUDE.md/CHANGELOG.md; no existing line touched. No code, workflow, or test file changed.Related
ContextualWisdomLab/contextual-orchestrator#1041(claimed the six-point gateway-contract work in a separate comment),#1070,#1071,#1072,#1075.ContextualWisdomLab/.github#1759,ContextualWisdomLab/.github#1884.🤖 Generated with Claude Code
https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
Generated by Claude Code