[agora-self-update] Prompt Tuning: agora-config-update routes implementation lessons to an AgentConfig no implementing agent loads #1734

🤖 Kelos Agent @gjkim42

Latest verdict

• Status: VALID

• Checked at: 2026-11-03T06:01:30Z

• Evidence: Kelos main moved from a6f6225 to d0ae450 through #1867 (shareable Console session URLs) and #1868 (Session subsessions). Neither touches self-development/ or internal/examples/self_development_test.go. The last commit touching self-development/ is still 3040ff1 (Oct 15), and b1e0398...main changes none of the cited files, so the permalinks below still match main. The prompt still has the problem:

  • In agora-config-update.yaml, line 65 still calls agentconfig.yaml "shared Agora role instructions".
  • Lines 93–99 still route "Shared changes (affect all Agora agents)" there.
  • Lines 115–121 still give no way to withdraw an unsupported change from the agent's own PR.
  • README line 253 repeats the wording.

  The wiring is unchanged and still pinned by internal/examples/self_development_test.go (lines 202–211):

  • agora-workers and agora-pr-responder (Codex, gpt-6.1-sol) load only base-agent.
  • agora-dev-agent is loaded only by agora-triage and agora-squash-commits.

  PR #1664 is still open and unassigned at 30a8a25. It still adds only a two-line docs rule to agentconfig.yaml, which the Documentation completeness check in agora-reviewer.yaml (line 214) already covers. It has had no activity since the Sept 24 comment, and is still the only open PR touching self-development/agora/. On the Agora side:

  • main is still 2f3ddab (#7, July 7), and there are no open PRs.
  • The only open issues are the unassigned fake-user slot #5 and fake-strategist slot #8. Both were refreshed in the last day.
  • Agora's Makefile targets (test, update, verify, build, build-cli, build-server, image, run, clean) still match what agentconfig.yaml lists.

  The Open Actions counterpart, #1626, is still open and unassigned. #766 still covers a different fleet. I rechecked the lower-ranked candidates, and none outranks this one:

  • The agora-reviewer sticky lookup (lines 329–333) still pairs --slurp with --jq, which gh 2.103.3 rejects. PR Add CLI tool for task management #7 still has a single sticky comment, so no Agora failure has been observed. The same lookup appears in reviewer files across all four fleets, so it does not fit an Agora-only slot.
  • Line 81 of the fake-user prompt still suggests go run ./cmd/agora.
  • Step 7 of agora-squash-commits (lines 120–122) still adds kelos/needs-input, which no Agora spawner filters on.

  The last two remain minor.

Area

Prompt Tuning

Problem and impact

agora-config-update is the only loop that turns Agora PR review feedback into agent configuration. Its prompt describes self-development/agora/agentconfig.yaml as "shared Agora role instructions" and routes "Shared changes (affect all Agora agents)" there. The fleet is not wired that way:

• agora-dev-agent (agentconfig.yaml) is loaded only by agora-triage and agora-squash-commits. Neither of them writes Agora code.
• The agents whose PRs get reviewed are the agora-workers and agora-pr-responder Sessions, which run Codex (gpt-6.1-sol). They load only base-agent, which the Codex entrypoint writes to $CODEX_HOME/AGENTS.md. They take Agora's project conventions from AGENTS.md in the kelos-dev/agora checkout, which Codex reads natively. This wiring is deliberate and pinned by internal/examples/self_development_test.go.
• The planner, reviewer, fake-user, and fake-strategist each load their own inline AgentConfig.

The map went stale on July 23, when 612303d moved agora-pr-responder from agora-dev-agent to base-agent only. Since then, a lesson about implementation output, which is the most common kind of review feedback, lands in a file that no implementing agent reads. The resulting PR looks like a fix but changes nothing for the agents that made the mistake. The prompt also has no path for withdrawing such a change once a run notices it, so the PR stays open.

Evidence from the workflows and agent activity

• PR #1664 is the only PR agora-config-update has ever opened. It has been open and unassigned since Aug 19, and was last pushed Aug 31. It adds a docs rule to agentconfig.yaml in response to Agora PR #7, and its description says "The shared rule lives in self-development/agora/agentconfig.yaml so all Agora implementation agents receive it." In fact the rule reaches only triage and squash-commits.
• The Sept 24 run commented on that PR that "the line added here never reaches the agents that left out the docs in Require AGORA_URL for server CLI commands agora#7". It then left the PR open and handed the decision to a maintainer. It said the SessionSpawner fix was "outside the self-development/agora/ scope this agent is allowed to edit", but both files are under self-development/agora/. The prompt gives the agent no accurate picture of which agent loads which config, and no allowed action once it finds its own PR is wrong.
• The PR Add CLI tool for task management #7 lapse was already caught where a check can reach it. The Documentation completeness check in agora-reviewer has existed since the initial port. It flagged the missing docs as P2, and the responder fixed them in the same cycle.
• Prompt: agora-config-update.yaml file map, lines 62–66, routing, lines 90–99, and existing-PR handling, lines 115–121.
• Wiring: agora-workers.yaml lines 56–57, agora-pr-responder.yaml lines 63–64, agora-triage.yaml lines 49–51, agora-squash-commits.yaml lines 59–61, and the test pins.
• The README's How It Works section describes the wiring correctly, but its config-update section, line 253 repeats the prompt's "shared Agora role instructions in agentconfig.yaml" wording.

Proposed change

In self-development/agora/agora-config-update.yaml:

1. Replace the agentconfig.yaml bullet in the file map (line 65) with:

   • self-development/agora/agentconfig.yaml: agora-dev-agent, loaded only by agora-triage and agora-squash-commits
   • The inline AgentConfigs in agora-planner.yaml, agora-reviewer.yaml, agora-fake-user.yaml, and agora-fake-strategist.yaml, each loaded only by its own spawner
   • The agora-workers and agora-pr-responder Sessions load only base-agent, and read Agora's project conventions from AGENTS.md in the kelos-dev/agora repository, which this agent does not edit

2. Replace the "Shared changes" and "Task-specific changes" blocks (lines 93–99) with routing by the agent that produced the output under review. Keep the base-agent skills rule.

   Route each lesson to the configuration that the agent behind the reviewed output actually loads:

   • Triage or squash-commits behavior: agentconfig.yaml
   • Planner, reviewer, fake-user, or fake-strategist behavior: that spawner's prompt or inline AgentConfig
   • Implementation output from agora-workers or agora-pr-responder (code, tests, docs, commits, PR descriptions): no file here reaches those Sessions. Do not add these rules to agentconfig.yaml, and do not expand the Sessions' minimal initialPrompt. If a review-time check would catch the problem and agora-reviewer.yaml does not already have it, add it there. Otherwise, make no change for that lesson.
   • Keep shared skills only in self-development/base-agent.yaml. Do not duplicate them in Agora role AgentConfigs.

3. Extend the existing-PR constraint (lines 115–121) with:

   If an existing unassigned PR from this agent contains changes that this routing does not allow, or that the evidence no longer supports, remove them. If no changes remain, close the PR with a comment explaining why.

Then update line 253 of self-development/agora/README.md to describe the same routing.

Acceptance criteria

• agora-config-update.yaml states which agents load agentconfig.yaml, which load each inline AgentConfig, and which load only base-agent. It no longer says agentconfig.yaml affects all Agora agents.
• Lessons are routed by the agent that produced the reviewed output. Rules for worker or responder implementation output never go to agentconfig.yaml. They go to agora-reviewer.yaml when a review-time check fits and is not already there.
• The agent may remove unsupported changes from its own unassigned PR, and close that PR when nothing remains. The assigned-PR guard, the one-PR-per-run rule, /kind cleanup, and release-note: NONE are unchanged.
• self-development/agora/README.md describes the same routing.
• agentConfigRefs do not change, so internal/examples/self_development_test.go needs no update. Only files under self-development/agora/ change, and make verify passes.
• Post-merge check: the next agora-config-update run closes PR #1664. Its only change adds an implementation rule to agentconfig.yaml, and agora-reviewer.yaml already has the matching check.

Related work and scope

• PR #1664 is the misrouted PR itself. This issue fixes the prompt that produced it, and closing Teach Agora agents to document public contract changes #1664 is the expected outcome rather than a separate change.
• #766 covers kelos-config-update and Kelos's inline AgentConfigs, a different fleet and a different gap.
• Adding agora-dev-agent to the two SessionSpawners is not proposed. The base-agent-only Session wiring is deliberate across all four fleets and pinned by the test. agora-dev-agent also carries ephemeral-container and bot-identity text that does not fit Sessions running on the maintainer's token with a persistent volume.
• Open question for the maintainer: should lessons that no review-time check can catch be proposed for Agora's AGENTS.md (for example, as a kelos-dev/agora issue)? That would mirror how kelos-config-update updates the Kelos AGENTS.md. This proposal leaves it out.
• The Kanon and Open Actions config-update prompts use the same "affect all … agents" wording. They are outside this issue's scope.
• Also considered, ranked lower:
  • The fake-user prompt's go run ./cmd/agora starts the CLI, not the server, and make build is described as "the agora binary" although it builds bin/agora and bin/agora-server. Agents fall back to make run.
  • The public-contract lists in triage, planner, and reviewer omit the agora CLI commands and flags. No triage, plan, or review run has happened since July, so no failure has been observed yet.