feat(ci): add evidence-gated issue-draft composer; add ADR-0022 - #1685
feat(ci): add evidence-gated issue-draft composer; add ADR-0022#1685seonghobae wants to merge 5 commits into
Conversation
The owner asked that Noema and/or the OpenCode Agent also handle PR follow-up work, process review feedback, search the web, find papers, handle issues, and author issues, to the same evidence-based, ADR-traceable standard as docs/product-goal-directive.md. Re-verified the given hard constraint (opencode.jsonc's edit:deny on the pull_request_target-triggered required review gates) is still true against a fresh clone before any design work. Investigation (re-verified, not assumed from a prior read) found: review/rca/conflict autofix already exists and needed no widening; contextual-orchestrator's web_search.py is real but unmerged with zero callers; academic paper search has no code anywhere; noema-core (PR #536, hours old) has no tool/edit/sandbox machinery and is not a viable host yet; and, critically, docs/product-goal-directive.md's standing autonomous-loop authorization is written entirely in PR terms and never mentions issue creation - a genuine authorization gap, not an assumption in either direction. ADR-0022 records the trust-boundary reasoning (the required gate and the autofix worker sit on opposite sides of the same boundary and stay there), the chosen architecture, and the full deferred roadmap (web search wiring, paper search, unattended issue-creation triggering, noema-core adoption), connecting to backlog item 5 and noema's ADR-0012. The one increment shipped for real: scripts/ci/issue_draft_composer.py, a pure evidence-gated composer that rejects a draft with no findings, citations, or traceable source, and only calls `gh api -X POST repos/{repo}/issues` when invoked with --create explicitly. No workflow wires --create into any trigger in this change, sidestepping the authorization gap rather than assuming it away. 100% coverage/docstrings on the new module, matching this repository's existing scripts/ci gate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Warning Review limit reachedNext included review available in 5 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| description = _non_empty_str(raw.get("description"), field=f"findings[{index}].description") | ||
| citation = _non_empty_str(raw.get("citation"), field=f"findings[{index}].citation") |
There was a problem hiding this comment.
🟡 Placeholder evidence passes the gate
load_draft accepts any nonblank citation and source, including c and z. --create can therefore publish an issue with no traceable evidence.
Prompt for agents
Strengthen scripts/ci/issue_draft_composer.py's load_draft evidence validation so source and every finding citation have a defined, mechanically traceable format rather than merely being nonblank. Decide and document the accepted forms, such as repository path plus line/range, SHA-pinned repository URL, issue/PR URL, DOI, or another durable source identifier. Add rejection tests for arbitrary placeholders such as "c" and "z", while retaining tests for each supported citation form.
Was this helpful? React with 👍 or 👎 to provide feedback.
| lines = ["## Summary", "", draft.summary, "", "## Evidence", ""] | ||
| for finding in draft.findings: | ||
| lines.append(f"- {finding.description} ({finding.citation})") | ||
| lines.extend(["", "## Source", "", draft.source, "", ATTRIBUTION_FOOTER]) |
There was a problem hiding this comment.
🟡 Oversized drafts fail at publication
render_markdown_body accepts unbounded evidence text. A body beyond GitHub's issue limit passes validation, then gh api rejects creation.
Prompt for agents
Add a GitHub issue-body size contract to scripts/ci/issue_draft_composer.py. Validate the final rendered body, including headings and the attribution footer, before invoking gh. Return an IssueDraftError with a clear limit message and add boundary tests for the largest accepted body and the first rejected size. Ensure main handles this validation in both draft and create modes.
Was this helpful? React with 👍 or 👎 to provide feedback.
| result = json.loads(run(args)) | ||
| return str(result.get("html_url") or "") |
There was a problem hiding this comment.
📝 Info: Creation failures remain visible
create_issue propagates command and response failures. The CLI exits nonzero rather than reporting a false success, and the shared runner scrubs diagnostics.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if not args.create: | ||
| print(render_draft_text(draft)) | ||
| return 0 | ||
|
|
||
| print(create_issue(draft)) |
There was a problem hiding this comment.
📝 Info: Dry-run boundary is effective
Draft mode returns before create_issue, and no workflow references this module. The PR therefore adds no unattended issue-creation path.
Was this helpful? React with 👍 or 👎 to provide feedback.
| from scripts.ci.pr_review_merge_scheduler import run | ||
|
|
||
|
|
||
| REPOSITORY_RE = re.compile(r"^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$") |
There was a problem hiding this comment.
🟨 Repository target permits traversal segments
REPOSITORY_RE accepts ../repo and owner/... Untrusted evidence can alter the REST endpoint instead of selecting one repository.
Was this helpful? React with 👍 or 👎 to provide feedback.
…independent The module's try/except ModuleNotFoundError import (the pattern most cross-referencing scripts/ci modules use to work both package-qualified and as a bare sibling import) only had its except branch exercised by accident of which test file happened to run first in the full suite and mutate sys.path - a stale worktree read had masked this behind a `# pragma: no cover` initially, and removing it exposed a genuine 98% gap (lines 34-35) when the full suite's actual collection order was checked instead of assumed. Added a dedicated test that clears the bare-name sys.modules cache entry and strips scripts/ci from sys.path before re-executing the module fresh under a private name, forcing the except branch deterministically regardless of what else has run. Registers the fresh module in sys.modules before exec so its dataclasses can resolve their (`from __future__ import annotations`-deferred) field types via sys.modules[cls.__module__], which importlib.util.module_from_spec does not do automatically the way a normal import statement does. Verified both orderings directly: standalone (tests/test_issue_draft_composer.py alone) and with tests/test_agent_mention_sweep.py collected first (the file whose sys.path mutation had been masking the gap) - both report 100% coverage on scripts/ci/issue_draft_composer.py now. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
| monkeypatch.delitem(sys.modules, "pr_review_merge_scheduler", raising=False) | ||
| monkeypatch.setattr(sys, "path", [p for p in sys.path if p != SCRIPTS_CI_DIR]) | ||
|
|
||
| spec = importlib.util.spec_from_file_location( | ||
| "issue_draft_composer_fallback_import_check", MODULE_PATH | ||
| ) | ||
| module = importlib.util.module_from_spec(spec) | ||
| # Register before exec: the module's dataclasses resolve their string annotations (from | ||
| # __future__ import annotations) via sys.modules[cls.__module__], which only exists for a | ||
| # normal `import` statement by default. | ||
| monkeypatch.setitem(sys.modules, spec.name, module) | ||
| spec.loader.exec_module(module) |
There was a problem hiding this comment.
📝 Info: Fallback test isolates interpreter state
monkeypatch restores the temporary sys.path and sys.modules changes. Registering the fresh module before execution also preserves dataclass annotation resolution.
Was this helpful? React with 👍 or 👎 to provide feedback.
seonghobae
added
area: ci-cd
ChatGPT Codex Connector
![cwl-noema-review[bot]](https://avatars.githubusercontent.com/u/295022177?s=60&v=4){kind=small}
There was a problem hiding this comment.
Noema LLM review
The PR implements the evidence-gated issue-draft composer as specified in ADR-0022, but it fails to address critical security and correctness issues identified in prior review threads. Specifically, the repository regex allows path traversal segments, the evidence gate is too permissive (accepting arbitrary placeholders), and there is no validation for the final rendered body size, which will lead to API failures for oversized drafts.
Reviewed changed lines
scripts/ci/issue_draft_composer.py:38 (RIGHT): REPOSITORY_RE allows dots and dashes in both segments, which permits traversal segments like '..' that could manipulate the REST API endpoint.scripts/ci/issue_draft_composer.py:110 (RIGHT): The evidence gate relies on _non_empty_str, which only checks for non-blank strings. This allows meaningless placeholders (e.g., 'z') to pass as valid citations/sources.scripts/ci/issue_draft_composer.py:134 (RIGHT): render_markdown_body does not enforce a maximum length. GitHub's issue body limit is not checked, meaning the tool will pass local validation but fail during the 'gh api' call.
Adversarial validation
scripts/ci/issue_draft_composer.py:38 (RIGHT)confirmed: The repository regex allows traversal segments. — REPOSITORY_RE = re.compile(r'^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$') allows dots in both segments.scripts/ci/issue_draft_composer.py:110 (RIGHT)confirmed: Placeholder evidence passes the gate. — _non_empty_str only checks if the string is not blank.- Residual risk: High. The tool is vulnerable to API endpoint manipulation via the repo field, allows the creation of issues with non-traceable evidence, and lacks a size contract for the issue body.
Findings
- [high] scripts/ci/issue_draft_composer.py:38 (RIGHT): Repository target permits traversal segments. REPOSITORY_RE should be tightened to prevent '..' or other segments that could alter the REST endpoint.
- [medium] scripts/ci/issue_draft_composer.py:134 (RIGHT): Missing GitHub issue-body size contract. The rendered body should be validated against GitHub's size limits before attempting to call the API.
- [medium] scripts/ci/issue_draft_composer.py:110 (RIGHT): Evidence validation is too weak. Citations and sources should follow a mechanically traceable format rather than just being non-blank.
- Result: REQUEST_CHANGES
- Head SHA:
2b06563238221a9cd4ecc776de5f0eff270bdc59 - Reviewer credential:
noema-review-github-app-refresh - Actor:
cwl-noema-review[bot]
# Conflicts: # CHANGELOG.md # docs/product-technical-gap-baseline.md
Merge-conflict repair (main advanced since this branch's base)
2 conflicting files, both ordinary append-only journal entries under the same dated-entry format — no semantic overlap, kept both sides:
No other file conflicted — this branch's actual payload ( Verification
Pushed non-force to the same branch: Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com Generated by Claude Code |
Both conflicts are purely additive and both sides are kept: - CHANGELOG.md — this branch's entry above main's, newest-first convention. - docs/product-technical-gap-baseline.md — both sides add a distinct top-level `## 2026-09-02 …` section at the same anchor. Unlike a sibling merge on this file, neither block continues the paragraph above it (both open with their own `##` heading), so concatenation does not orphan any prose. Verified after resolution that the section sequence reads correctly (2629 sidecar pool / 2652 this branch's agent-PR scope / 2779 main's org-queue-sweep / 2818 stale cron) and that fenced code blocks stay balanced. Verified after resolution: 2919 passed, 1 skipped, coverage 100%, interrogate 100%; zero conflict markers; `ruff check --select F821` clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 participants
Summary
The owner asked that Noema and/or the OpenCode Agent also handle PR follow-up work more broadly,
process review feedback, search the web, find papers, handle issues, and author issues on the
owner's behalf, to the same evidence-based, ADR-traceable standard
docs/product-goal-directive.mdalready holds PR work to. The hard constraint given alongside the ask — the required,
pull_request_target-triggered review gates (opencode-review.yml,noema-review.yml) must keep"edit": "deny"— was independently re-verified true against a fresh clone before any design workstarted, and nothing in this PR touches that file or that trigger.
The required gate (judges first-look, unauthenticated-author content) and the existing
pr-review-autofix.ymlworker (acts only on already-reviewed content,repository_dispatch-only,scoped edit allowlist, mandatory fresh re-review of its output) sit on opposite sides of the same
trust boundary and stay there.
noema-coreis not chosen as a host for anything in this pass — ithas no tool/edit/sandbox machinery yet (PR feat(reviewer): extract shared noema-core package (ADR-0014) noema#536, hours old) — but the
shipped increment is deliberately pure/side-effect-free so a future move there (once ADR-0012
grows tool machinery) is a lift, not a rewrite. Connects to backlog item 5 (Noema as a reusable DDD
agent for naruon) and noema's own ADR-0012.
docs/product-technical-gap-baseline.md(web search wiring once
contextual-orchestrator#1009merges, an academic paper-search client,wiring
--createinto a trusted trigger once the directive names issue-creation authorization,and eventual
noema-coreadoption).scripts/ci/issue_draft_composer.py— a pure,evidence-gated GitHub issue-draft composer. Rejects a draft with no findings, no citations, or no
traceable source. Renders the draft to stdout by default; only calls
gh api -X POST repos/{repo}/issueswhen invoked with--createexplicitly. No workflow inthis PR wires
--createinto any scheduled or dispatched trigger — a fresh re-read ofdocs/product-goal-directive.mdfound it authorizes autonomous PR work but never mentions issuecreation (zero occurrences of "이슈"/"issue" in its 96 lines), a genuine authorization gap flagged
explicitly rather than assumed away in either direction.
Why this is safe
opencode.jsoncis unmodified:edit: denystays on every agent, andthis PR adds no MCP/webfetch/websearch surface anywhere.
.github/workflows/calls it. It isa CLI a human (or an interactively-working agent) can invoke; unattended use is explicitly
deferred per ADR-0022 until the standing directive names issue-creation authorization.
create_issue's only side effect is a single, explicitgh api POST repos/{repo}/issuescall,gated behind
--create, using the samerun()/argv-list (no shell) pattern already usedthroughout
scripts/ci.Developer experience
scripts/ci/issue_draft_composer.py+tests/test_issue_draft_composer.py(28 tests): evidence-gaterejection for every required field (missing/blank repo, title, summary, source; malformed repo
slug; oversized title; non-list/empty findings; findings missing description or citation;
non-list/malformed labels), Markdown rendering content (summary/evidence/citations/source/
attribution footer), the
--create-gatedgh apiargv (including repeatedlabels[]fields) via amonkeypatched
run()— the same seampr_review_fix_scheduler.py's own tests use — the CLI'sdraft-only default (asserts
run()is never called), error paths (bad JSON, missing file, failedevidence gate all exit 1 with a clear stderr message), and the
__main__guard viarunpy.Full-repository
coverage run -m pytest testswas also started locally to confirm nothing elseregressed; this machine had several other concurrent, unrelated agent sessions consuming CPU at the
same time, so it had not finished by the time this PR was opened. No existing file was modified
except two docs (
CHANGELOG.md,docs/product-technical-gap-baseline.md) and no existing scriptchanged, so the blast radius against the existing suite is structurally limited to test collection
succeeding — which the standalone run already confirms for the new file. CI will run the
authoritative hash-pinned full suite on this PR.
User experience
No change for any PR author or reviewer: no new workflow trigger exists, so no issue can appear on
any repository as a side effect of this PR.
Deferred (full detail in ADR-0022 and the gap-baseline entry)
contextual-orchestrator#1009on its own review, then wire it intopr-review-autofix.yml's generated config with a new outbound-scope check (owner:contextual-orchestrator, then.github).contextual-orchestratorornoema-coreonce tool-capable, then.github).--createinto a trusted, scoped trigger oncedocs/product-goal-directive.mdexplicitlyauthorizes unattended issue creation (owner:
.github, blocked on a directive-text decision).noema-coreas an@agent.toolonce it growstool/edit/sandbox machinery (owner:
noema)..github#1682merges its "immature core" directive text.🤖 Generated with Claude Code