Add programmatic memory harness - #61
Conversation
Reviewer's GuideIntroduces an experimental Sequence diagram for programmatic_memory decision and bounded retrievalsequenceDiagram
actor QP as QuestPlayer
participant PMH as ProgrammaticMemoryHarness
participant TCH as ToolCompactHarness
participant M as Model
participant TRAJ as Trajectory
QP->>PMH: get_action(observation, choices)
PMH->>TCH: get_action(observation, choices)
TCH->>M: tool-select prompt (DefaultMemory)
M-->>TCH: JSON response
TCH->>PMH: parsed LLMResponse
PMH->>PMH: _extract_tool_calls(response)
PMH->>PMH: _execute_tool_calls(tool_calls[:1])
alt history_read
PMH->>PMH: history_read(start_step, count)
PMH->>TRAJ: read(start_step, count)
TRAJ-->>PMH: bounded range text
else history_search
PMH->>PMH: history_search(query, limit)
PMH->>TRAJ: search(query, limit)
TRAJ-->>PMH: bounded search text
else calculator/scratchpad
PMH->>PMH: calculator()/scratchpad()
end
PMH-->>TCH: tool_results
TCH->>M: final action prompt
M-->>TCH: final JSON with result
TCH-->>PMH: selected action
PMH->>TRAJ: append(observation, choices, selected_action, selected_choice)
PMH-->>QP: action index
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Important Review available on request
Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e7f6586169
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e4abc0ff77
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b52d91bedf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 17c37725c5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| grouped: dict[str, list[RunInsight]] = defaultdict(list) | ||
| for insight in insights: | ||
| grouped[insight.model].append(insight) | ||
| grouped[_group_label(insight.model, insight.harness, harnesses_by_model)].append(insight) |
There was a problem hiding this comment.
When the new three-quest pilot is run, this grouping assigns every run solely to its model/harness row, and calculate_summary_stats() does the same, so neither standard report exposes per-quest success rates. This directly prevents the required per-quest analysis in docs/PROGRAMMATIC_MEMORY_PROPOSAL.md:224; an aggregate improvement driven only by an easy quest will appear as a harness-wide gain. Add a quest-by-harness breakdown (or nest quest statistics beneath each harness) before interpreting the pilot.
Useful? React with 👍 / 👎.
1 participant
Summary
programmatic_memoryharness, prompt, registry wiring, and pilot benchmark matrixVerification
uv run pytest llm_quest_benchmark/tests/harnesses/test_trajectory.py llm_quest_benchmark/tests/harnesses/test_harnesses.py llm_quest_benchmark/tests/harnesses/test_factory.py llm_quest_benchmark/tests/integration/test_mode_agents_e2e.py llm_quest_benchmark/tests/test_database.py(96 passed, 3 skipped)uv run ruff format ...(7 files unchanged)uv run ruff check ...(pass)Boat.qmfake-provider smoke: 18 steps, persistedhistory_search, terminal FAILURE without runtime errorThe live-provider pilot is intentionally not run in this PR.
Summary by Sourcery
Introduce an experimental programmatic-memory harness with deterministic trajectory retrieval, bounded quest execution, and end-to-end benchmark integration.
New Features:
programmatic_memoryharness with deterministic full-trajectory history reads and searches, calculator and scratchpad tools, prompt wiring, and benchmark registration.Bug Fixes:
Enhancements:
Documentation:
Tests: