Proposal: Automated PR Review with Raft Agents #4023
Replies: 2 comments
|
Thanks for the detailed proposal — the end-to-end Raft validation and the clear service/agent boundary (credentials never reaching agents, strict structured-output validation) show real care, and the problem it targets is real. That said, I'd like to question the overall shape before we invest in it: most of what the service implements already runs today, and the one missing piece is much smaller than a service. Our current Raft review setup (the The effort labels we added yesterday also already solve the prioritization problem the eligibility policy encodes: So the only genuinely missing piece is the trigger: today a human has to @mention the orchestrator. That gap can be closed with a thin scheduled script (a GitHub Action or cron) that runs your eligibility rules and posts I'd suggest we start with that thin trigger, run it for a while, and only reach for a dedicated service if the simple version demonstrably breaks (e.g. trigger volume or bookkeeping outgrows the channel). The eligibility policy section of this proposal transfers directly to the script, so that design work is not wasted. One process note: since this is an architecture decision, let's make sure the conclusion lands on dev@ — the Discussion sync covers that, so discussing here works. 简体中文感谢这份详细的提案——端到端的 Raft 验证和清晰的服务/agent 边界（凭据不进 agent、严格的结构化校验）都做得很扎实，要解决的问题也是真实的。 不过在投入之前，我想先质疑一下整体形态：服务要实现的大部分能力，现在已经在跑了；真正缺的那一块比一个服务小得多。 现有的 昨天加的 effort label 也已经解决了准入规则想解决的优先级问题： 所以真正缺的只有触发：现在需要人 @ 编排者。这一步用一个薄的定时脚本（GitHub Action 或 cron）就能补上：跑你设计的准入规则，把合格 PR 以 建议先上这个薄触发器跑一段时间，只有当简单版本被证明撑不住时（比如触发量或簿记超出频道承载）再考虑专用服务。提案里的准入规则部分可以直接迁移到脚本里，这部分设计不会浪费。 流程上补一句：这属于架构决策，结论要落到 dev@ ——Discussion 会自动同步，所以在这里讨论没问题。 |
Revised Direction: A Thin Automatic Trigger for the Existing Raft Review WorkflowThanks for the detailed feedback. I agree with the central point: the existing What changes from the original proposalThe following components are removed:
Those responsibilities remain with the existing Raft review workflow, which The new component has only two responsibilities:
Initial automatic-review policyA PR head is eligible only when all of the following are true:
The classifier uses an OpenAI-compatible API. Its base URL and model name are Large PRs, End-to-end flowThe message should be intentionally small and should not attempt to prescribe Recent Polling first, webhook laterThe personal prototype will run a scheduled scan on a server. Polling is used The trigger keeps only a small durable ledger keyed by If the community later adopts the trigger as official Maka infrastructure, the Deployment and ownershipThe first version is one small process or cron job with configuration for:
It needs GitHub read access, a Raft identity that can post in the target The prototype should remain in a personal repository while the behavior is Non-goals
This version is deliberately a thin extension of the current workflow. A more 简体中文修改方向：为现有 Raft Review 流程增加一个薄的自动触发器感谢详细的反馈。我认同其中最关键的判断：现有 相比原提案删除的内容新方案删除：
这些职责继续由现有 Raft Review 流程承担。它已经提供多线 Review、exact-head 处理、去重、失败重派、结论合成、GitHub 发布和自动化披露。 新组件只负责两件事：
自动准入规则只有同时满足以下条件的 PR Head 才会自动触发：
Classifier 使用 OpenAI-compatible API，Base URL 和 Model Name 均通过配置提供。它只允许返回 大型 PR、 完整流程触发消息保持简单，不指定 Review Lane 或具体 Reviewer：
先轮询，后 Webhook个人原型第一版部署在服务器上定时扫描。当前使用轮询，是因为它不需要修改 触发器只保存一个以 如果社区以后将它接纳为 Maka 正式基础设施，推荐改成 GitHub Webhook 实时通知，并保留定时扫描用于对账。在此之前，轮询可以保持个人实验与 Apache 仓库基础设施之间的边界。 部署与归属第一版只是一个小型常驻进程或 Cron Job，配置内容包括：
它需要 GitHub 只读访问、可以在目标 Channel 发消息的 Raft Identity，以及用于分类的 LLM Credential。它不需要 GitHub 写凭据，因为现有 Raft 流程仍是唯一的发布链路。 验证阶段应继续放在个人仓库。以后如果 Maka 社区决定接纳并负责运行它，再通过正常贡献流程把这个小触发器放入 非目标
该版本有意保持为现有流程上的薄扩展。只有实际触发量或运行故障证明它无法满足需求时，才考虑更复杂的服务。 |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Summary
This proposal adds automatic review for small Maka bug-fix pull requests.
A small service watches GitHub, decides whether a pull request is safe and
small enough for automatic review, and selects an available Raft agent. The
agent reviews the exact PR commit and returns structured findings. The service
then publishes those findings as GitHub review comments using a shared bot
account.
The service handles scheduling and GitHub access. Raft remains responsible for
running the review agents.
Background
Small bug-fix PRs still require time from Maka members even when the change is
limited and the CI result is clear. Maka members already use Raft agents for
code review, but that workflow starts with a person manually asking an agent to
review a PR.
This proposal automates the repeatable part of that workflow. It does not try to
automate every review or replace maintainers. It gives eligible small fixes a
consistent first review while keeping GitHub credentials and scheduling rules
outside the agents.
Why not use a supervisor agent for everything?
A Raft agent is a good fit for reading code and finding problems. It is not a
good place to keep durable scheduling state or GitHub credentials.
The review service therefore owns the deterministic work:
Raft agents only do the part that needs reasoning: reviewing the code.
Scope
The first version starts with the following default eligibility policy. These
values are proposed starting points and can be adjusted after the service is
used in practice. A PR is automatically reviewed only when all conditions are
true:
effort/XSoreffort/Slabel;Large fixes,
featPRs,refactorPRs, drafts, and conflicting PRs are notautomatically reviewed.
Manual review is outside this service. A Maka member can still ask an agent
directly in Raft when an extra review is needed.
Fix classification
The service does not infer PR type from a title prefix. It sends exactly one
classification request containing only the PR title and description to an
OpenAI-compatible Chat Completions API.
The classifier returns one of three values:
{ "classification": "fix", "reason": "The PR corrects incorrect existing behavior." }classificationmust befix,not_fix, oruncertain. Onlyfixpasses theautomatic-review gate. An invalid response, timeout, or API failure is treated
as
uncertainand does not start a review.Results are cached by a hash of the title, description, and classifier prompt
version. A later push does not cause another query when those inputs are
unchanged. The stored audit record includes the decision, reason, model name,
and prompt version.
The API endpoint and model are configured without changing the code:
The API key itself is read from the named environment variable and is never
stored in the configuration file.
End-to-end flow
GitHub events
The service reacts to PR creation and later pushes through GitHub webhooks. It
also listens for CI completion and changes that may affect mergeability.
Webhooks are treated as notifications, not as commands. After each relevant
event, the service fetches the latest PR state from GitHub and evaluates the
same rules again.
The initial reconciliation interval is ten minutes. The job catches missed
webhooks and tasks left incomplete by a temporary failure.
When a new commit is pushed to a PR, unfinished work for the old head becomes
stale. The new head must pass the rules again before a new task is created.
Raft channel and reviewer pool
Automatic reviewers and one Dispatcher identity are placed in a private Raft
channel such as
#maka-auto-review.The channel has three jobs:
@reviewermessage that wakes an agent.The service configuration decides which agents can actually receive work. An
agent is eligible only when it is enabled, matches the changed module, is in the
review channel, has completed App Login, and can call the review Actions.
When channel membership changes, an operator updates this configuration. Being
present in the channel alone is not enough because the agent may not have
completed App Login or may not support Action calls.
Reviewer selection
Each agent may handle at most one automatic review at a time.
The service first keeps only agents that are enabled, match the changed module,
and have no active review. It then chooses the agent that has waited the longest
since its last assignment. A stable agent ID breaks ties.
If no matching agent is free, the task stays queued.
This scheduling code is deterministic. The Dispatcher does not use an LLM to
guess workload from channel messages.
Dispatcher
The Dispatcher is a Raft External Agent identity controlled by the review
service. It does not need an LLM or an agent runtime.
For each assignment, the service creates a Raft Task containing only a short
reference:
The explicit mention wakes the reviewer, and the reviewer claims the Task.
Changing a Task owner is not used as the wake-up mechanism.
Raft provides the visible task record. The full PR data, assignment state, and
submitted results remain in the review service.
Raft App Actions
The review service is registered once as a private Raft App. Its manifest
exposes two HTTP Actions:
get-review-tasksubmit-reviewThis does not copy or inject custom code into every agent. A compatible Raft
runtime discovers the registered Actions, and each reviewer completes App Login
once before using them.
The manifest uses the current Raft HTTP Action contract:
The Agent Login callback must create a service session cookie that can be used
for both Action URLs. The service uses that session to identify the calling
Raft agent.
get-review-taskRequest:
{ "task_id": "task_01ABC" }Response:
{ "schema_version": "1", "task_id": "task_01ABC", "repository": { "full_name": "apache/maka", "clone_url": "https://github.com/apache/maka.git" }, "pull_request": { "number": 1234, "url": "https://github.com/apache/maka/pull/1234", "title": "fix(runtime): handle an empty response", "body": null, "base_sha": "base-sha", "head_sha": "head-sha" }, "review_requirements": { "policy": "maka-adversarial-review", "instructions": "Review this exact diff and report actionable defects.", "output_schema_version": "review-result.v1", "max_results": 10 } }Only the assigned reviewer may retrieve the task. Repeated calls from that
reviewer return the same data.
submit-reviewRequest:
{ "task_id": "task_01ABC", "head_sha": "head-sha", "results": [ { "severity": "P1", "title": "The error is silently ignored", "body": "Return the original error instead of reporting success.", "location": { "path": "src/runtime/example.ts", "side": "RIGHT", "line": 42 } } ] }Each item in
results[]contains:severity:P1,P2, orsuggestion;title: a short finding title;body: the explanation and expected fix;location.path: the changed file;location.side:RIGHTfor the new file orLEFTfor the old file;location.line: the exact diff line;start_sideandstart_linefor a multi-line comment.An empty
results[]means that the agent found no actionable problem.The service rejects the whole submission when the task, head SHA, file, line,
or result shape is invalid. It never turns an invalid inline comment into a
general comment silently.
Publishing the GitHub review
GitHub write credentials exist only in the review service. Review agents and
Maka members never receive them.
The service publishes one GitHub
COMMENTreview for the task. Its validatedresults[]become inline comments in Files changed.Each comment ends with a short source marker:
The visible marker names only the Raft agent that produced the finding. The
GitHub account itself is a shared service account.
If
results[]is empty, the service publishes one overall comment saying thatthe agent found no actionable issue. Automatic review never submits
APPROVEor
REQUEST_CHANGES.After accepting a valid submission, the service publishes it without running
the complete eligibility check again.
Task state and retries
There is at most one review task for a repository, PR number, and head SHA.
Initial default retry rules:
get-review-taskwithin 10 minutes, reassignthe task;
After reassignment, the previous agent can no longer retrieve or submit the
task.
The service deduplicates webhook deliveries and uses a SQLite outbox for Raft
messages and GitHub publication. A hidden task marker in the GitHub review
prevents duplicate comments after an uncertain retry.
Deployment
The first version is one TypeScript application with one persistent SQLite
database:
The code is divided by responsibility:
github: GitHub state, diffs, webhooks, and review publication;policy: PR eligibility and path-to-module mapping;tasks: task state, scheduling, and retries;raft: reviewer pool, Dispatcher messages, App Login, and Actions;storage: SQLite transactions and the outbox.The first version does not need PostgreSQL, Redis, Kafka, microservices, or a
generic executor framework.
Raft validation completed
The Raft integration was tested end to end in an isolated channel with
synthetic PR data and no GitHub access. A real managed
test-agentsuccessfully:results[];accepted: true;test-agentin the service's submission record.The test confirmed that the proposed Raft App and Action path is usable by a
real Raft review agent.
Feedback wanted
The proposal is especially looking for feedback on these points:
get-review-taskandsubmit-reviewprovide enough information for theexisting Raft review agents?
Non-goals
feat,refactor, draft, or conflicting PRs;References
点击展开中文
提案：使用 Raft Agent 自动 Review PR
摘要
本提案为 Maka 的小型 Bug Fix PR 增加自动 Review。
一个独立服务负责监听 GitHub、判断 PR 是否适合自动 Review，并选择当前空闲的 Raft Agent。Agent Review 当前 PR 提交后，把结构化结果交还服务。服务再使用统一的 GitHub Bot 账号发布行级 Review Comment。
服务负责调度和 GitHub 操作，Raft 继续负责运行 Review Agent。
背景
即使改动范围很小、CI 结果明确，小型 Bug Fix PR 仍然需要 Maka Member 投入时间 Review。目前 Maka Member 已经会使用 Raft Agent Review 代码，但通常需要先由人手动请求某个 Agent Review 指定 PR。
本提案只把其中可重复的部分自动化，不尝试自动处理所有 Review，也不替代 Maintainer。它为符合条件的小型 Fix 提供稳定的第一轮 Review，同时把 GitHub Credential 和调度规则留在 Agent 之外。
为什么不全部交给主管 Agent？
Raft Agent 适合阅读代码和判断问题，但不适合保存长期调度状态或持有 GitHub 写凭据。
因此，Review 服务负责确定性的工作：
Raft Agent 只负责真正需要推理的部分：Review 代码。
范围
第一版先采用下面这组默认准入规则。这些数字只是初始值，可以根据实际运行情况调整。只有同时满足以下条件，服务才会自动 Review：
effort/XS或effort/SLabel；大型 Fix、
feat、refactor、Draft 和有冲突的 PR 不会自动进入 Review。本服务不提供手动入口。Member 如需额外 Review，仍可直接在 Raft 中请求 Agent。
Fix 分类
服务不再根据 Title Prefix 判断 PR 类型，而是向 OpenAI-compatible Chat Completions API 发起一次分类请求。输入严格限制为 PR Title 和 Description。
Classifier 返回以下三种结果之一：
{ "classification": "fix", "reason": "The PR corrects incorrect existing behavior." }classification只允许fix、not_fix或uncertain。只有fix可以继续进入自动 Review。输出格式错误、请求超时或 API 调用失败时统一视为uncertain，不启动 Review。服务根据 Title、Description 和 Classifier Prompt Version 计算 Hash 并缓存结果。后续 Push 如果这些输入没有变化，不会重复调用 LLM。审计记录保存分类结果、理由、Model Name 和 Prompt Version。
API Endpoint 和 Model 通过配置文件指定：
API Key 从配置指定的环境变量读取，不写入配置文件。
完整流程
GitHub 事件
服务通过 GitHub Webhook 监听 PR 创建和后续 Push，同时监听 CI 完成和可能影响冲突状态的变化。
Webhook 只表示“状态发生了变化”，不会直接命令服务开始 Review。每次收到相关事件后，服务都重新读取 GitHub 上的最新 PR 状态，再执行同一套准入检查。
第一版每十分钟执行一次对账，补偿遗漏的 Webhook 和因临时故障中断的任务。
PR Push 新 Commit 后，旧 Head 尚未完成的任务会变为 Stale。新 Head 重新通过准入检查后才会创建 Task。
Raft Channel 与 Reviewer 池
参加自动 Review 的 Agent 和一个 Dispatcher Identity 放在专用 Private Channel 中，例如
#maka-auto-review。Channel 负责三件事：
@Reviewer消息唤醒 Agent。服务配置决定哪些 Agent 可以真正接收任务。Agent 必须已启用、匹配修改模块、位于 Review Channel、完成 App Login，并且能够调用 Review Actions。
Channel 成员变化后，Operator 同步修改服务配置。只加入 Channel 并不足以接收任务，因为 Agent 可能尚未完成 App Login，或者 Runtime 不支持 Action 调用。
Reviewer 选择
每个 Agent 同时最多执行一个自动 Review。
服务先筛选已启用、模块匹配且当前空闲的 Agent，然后把任务交给最久没有收到任务的 Agent。时间相同时，使用稳定 Agent ID 打破平局。
没有匹配的空闲 Agent 时，Task 继续排队。
这部分是普通调度代码。Dispatcher 不使用 LLM，也不从 Channel 消息中猜测 Agent 负载。
Dispatcher
Dispatcher 是由 Review 服务控制的 Raft External Agent Identity，不需要连接 LLM 或 Agent Runtime。
每次分配时，服务创建一个只包含简短引用的 Raft Task：
明确的 Mention 会唤醒 Reviewer，随后 Reviewer 主动 Claim Task。修改 Task Owner 不作为唤醒方式。
Raft Task 提供可见的任务记录。完整 PR 信息、分配状态和 Review 结果仍保存在 Review 服务中。
Raft App Actions
Review 服务只需注册一次 Private Raft App，并通过 Manifest 暴露两个 HTTP Actions：
get-review-tasksubmit-review这不会向每个 Agent 复制或注入一份自定义代码。兼容的 Raft Runtime 可以发现已注册 Actions，每个 Reviewer 在第一次使用前完成一次 App Login。
Manifest 使用当前 Raft HTTP Action 协议：
Agent Login Callback 需要建立可以访问两个 Action URL 的 Service Session Cookie。服务通过这个 Session 识别调用它的 Raft Agent。
get-review-task请求：
{ "task_id": "task_01ABC" }响应：
{ "schema_version": "1", "task_id": "task_01ABC", "repository": { "full_name": "apache/maka", "clone_url": "https://github.com/apache/maka.git" }, "pull_request": { "number": 1234, "url": "https://github.com/apache/maka/pull/1234", "title": "fix(runtime): handle an empty response", "body": null, "base_sha": "base-sha", "head_sha": "head-sha" }, "review_requirements": { "policy": "maka-adversarial-review", "instructions": "Review this exact diff and report actionable defects.", "output_schema_version": "review-result.v1", "max_results": 10 } }只有当前被分配的 Reviewer 能够读取 Task。同一 Reviewer 重复调用时得到相同内容。
submit-review请求：
{ "task_id": "task_01ABC", "head_sha": "head-sha", "results": [ { "severity": "P1", "title": "The error is silently ignored", "body": "Return the original error instead of reporting success.", "location": { "path": "src/runtime/example.ts", "side": "RIGHT", "line": 42 } } ] }results[]中的每一项包含：severity：P1、P2或suggestion；title：简短标题；body：问题说明和修改建议；location.path：修改文件；location.side：新文件使用RIGHT，旧文件使用LEFT；location.line：对应的 Diff 行；start_side和start_line。空
results[]表示 Agent 没有发现需要指出的问题。如果 Task、Head SHA、文件、行号或 Result 格式不正确，服务会拒绝整个 Submission。服务不会悄悄把无效的 Inline Comment 改成普通 Comment。
发布 GitHub Review
GitHub 写凭据只保存在 Review 服务中，不会提供给 Review Agent 或 Maka Member。
服务为每个 Task 发布一个 GitHub
COMMENTReview。通过校验的results[]会成为 Files changed 页面中的 Inline Comment。每条 Comment 末尾包含简短来源：
可见来源只显示产出 Finding 的 Raft Agent，GitHub 发布账号使用统一的服务账号。
如果
results[]为空，服务发布一条总体 Comment，说明 Agent 没有发现需要指出的问题。自动 Review 不会提交APPROVE或REQUEST_CHANGES。服务接受有效 Submission 后直接发布，不在发布前重新执行完整准入检查。
Task 状态与重试
同一个 Repository、PR Number 和 Head SHA 最多只有一个 Review Task。
第一版默认重试规则：
get-review-task，重新分配；Task 重新分配后，旧 Reviewer 不能继续读取或提交。
服务对重复 Webhook 去重，并使用 SQLite Outbox 重试 Raft 消息和 GitHub 发布。GitHub Review 中的隐藏 Task Marker 用于防止不确定重试产生重复 Comment。
部署
第一版是一个 TypeScript 应用和一个持久化 SQLite 数据库：
代码按职责划分：
github：GitHub 状态、Diff、Webhook 与 Review 发布；policy：PR 准入和 Path-to-Module 映射；tasks：Task 状态、调度与重试；raft：Reviewer Pool、Dispatcher 消息、App Login 与 Actions；storage：SQLite Transaction 与 Outbox。第一版不需要 PostgreSQL、Redis、Kafka、微服务或通用 Executor Framework。
已完成的 Raft 验证
Raft 接入已经在隔离 Channel 中使用合成 PR 数据完成端到端测试，全程没有访问 GitHub。真实 Managed
test-agent成功完成：results[]；accepted: true；test-agent。这次测试证明，真实 Raft Review Agent 可以使用本方案中的 Raft App 和 Action 链路。
希望讨论的问题
希望社区重点反馈以下问题：
get-review-task和submit-review是否已经为现有 Raft Review Agent 提供了足够的信息？非目标
feat、refactor、Draft 或有冲突的 PR；参考资料
All reactions