Skip to content

docs: append cross-session agent know-how on nim_benchmark date gates and review sidecar gap - #1076

Draft
seonghobae wants to merge 1 commit into
mainfrom
docs/cross-session-agent-knowhow
Draft

docs: append cross-session agent know-how on nim_benchmark date gates and review sidecar gap#1076
seonghobae wants to merge 1 commit into
mainfrom
docs/cross-session-agent-knowhow

Conversation

@seonghobae

@seonghobae seonghobae commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Documentation-only. Appends two new AGENTS.md sections (with a pointer bullet in CLAUDE.md):

Recurring bug class: hardcoded review-cadence dates. nim_benchmark.py's fail-closed evidence gates (ACTUAL_COST_EVIDENCE["valid_until_date"], and any pricing-scenario's own valid_until_date) are deliberately literal calendar dates meant to lapse and force a human re-review — never "fix" a lapsed date by rewriting the production literal, that defeats the gate. This exact bug class has now bitten three separate test files this cycle (tests/test_nim_benchmark.py, tests/test_spend_analytics.py, tests/test_nim_benchmark_release_acceptance.py#1070/#1071/#1075). The established fix pattern is documented: an opt-in, not autouse, current_actual_cost_evidence fixture requested by name only from tests that need to get past the gate.

Central review sidecar/egress gap (tracked, not yet closed). scripts/ci/contextual_orchestrator_review_sidecar.sh (vendored by .github's four review-pipeline consumers) still injects all five raw provider secrets, clones and builds this repo fresh on the calling runner per invocation, and runs discovery in-process there. Citations: #1041 comment 5550412102 (six requirements for the immutable release this needs), ContextualWisdomLab/.github#1759.

See the companion ContextualWisdomLab/.github PR for the shared, org-wide half of this same effort (Codex-liveness, the GitHub Project-protocol MCP capability gap, and the @openai/codex CLI-invocation finding).

Verification

  • python -m pytest tests/test_nim_benchmark.py tests/test_nim_benchmark_release_acceptance.py tests/test_conventions.py -q and python -m interrogate -f 100 contextual_orchestrator/nim_benchmark.py both pass clean on this diff.
  • git diff against main is strictly additive to AGENTS.md/CLAUDE.md/CHANGELOG.md; no existing line touched. No code, workflow, or test file changed.

Related

  • ContextualWisdomLab/contextual-orchestrator#1041 (claimed the six-point gateway-contract work in a separate comment), #1070, #1071, #1072, #1075.
  • ContextualWisdomLab/.github#1759, ContextualWisdomLab/.github#1884.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX


Generated by Claude Code

… and review sidecar gap

Append-only additions to AGENTS.md (two new sections), a one-line pointer in
CLAUDE.md, and a CHANGELOG.md entry, recording lessons independently
re-verified against this exact checkout and live GitHub state on 2026-09-05:
the nim_benchmark.py evidence-gate recurring bug class (hardcoded
valid_until_date literals causing silent branch-coverage loss once lapsed,
diagnosed/repaired three times this cycle across PR #1070, PR #1071, and
issue #1075), and the still-open central review sidecar/egress gap tracked
in issue #1041 comment 5550412102 and ContextualWisdomLab/.github issue
#1759, including the PR #1884 conflation-then-correction cautionary tale.
Documentation only: no code, test, or behavior change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae seonghobae added documentation Improvements or additions to documentation priority: medium Normal-priority or P2 work type: docs Documentation, ADR, PRD, or technical writing labels Sep 7, 2026 — with ChatGPT Codex Connector
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation priority: medium Normal-priority or P2 work type: docs Documentation, ADR, PRD, or technical writing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants