Repository navigation
Environment interface (env-v1), Benchmark v1, and intervals in decision_stats - #79
Merged
Merged
Conversation
added 3 commits
November 8, 2026 02:27
environment.py: an AgentPort per chef. observe() and act() are plain dictionaries, the contract any agent can rely on: the kitchen state as the engine has it in English without display-only fields, every listed action unpruned, events since a sequence number, and act() checked when it arrives with an optional job guard. view is the engine read-only: every engine input (marked by the engine_input decorator) is refused, so all changes go through the port and land in the round's recorded inputs. fork_until_step_end exposes the isolated prediction ask_ahead uses. Jeff's DecisionLoop reads the kitchen through the view and starts actions and sprints through the port; payload and behaviour fingerprints are byte for byte unchanged. agent_loop.AgentLoop runs any environment.Agent in the game's session when the client factory returns one; example_agent.PriorityAgent plays whole rounds on observe() and act() alone, imports nothing of Jeff's, and its bundles replay exactly. environment.py and agent_loop.py ship with the runtime, and a test now checks that every project module a runtime file imports is in RUNTIME_FILES.
benchmark/v1/manifest.json freezes levels 1-3 (configuration hash and the hash of each golden reference trace), the scripted assembler partner (reference_policy.py), 12 rounds per level on a real clock and env-v1; a test fails when the code no longer matches it. scripts/benchmark.py runs every cell for Jeff on the saved connection or any environment.Agent (module:Class), in real time and in separate processes, keeping each round's bundle and its replay check; result.json reports money and goal rate with 95% intervals, the gain over the scripted chef alone and the agent's share of the work. verify refuses a result made for another manifest, from a checkout that no longer matches it, with the wrong number of rounds, on a virtual clock or with a round that does not replay; report puts results side by side; --dry-run prices a run. Docs: docs/architecture/environment-api.md and benchmark/README.md (English, then Chinese), pointers from both READMEs and docs/agent-integration.md, changelog entries. model_rounds reports each round's configuration hash and model.
A new uncertainty section gives 95% intervals for the goal rate, the share of decisions that chose the top option and of steps replaced by Jeff's own next choice (Wilson), and for the mean money, cooperation gain and Jeff's standard-time share per round and the median step gap per request policy (seeded bootstrap, research_stats.py). Every other field is unchanged; run from the same place on the same journals, decision-stats-17 and -18 agree on all of them, with and without --by-round and --split.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three commits on top of main (3ca48a0): review item ③ and the last part of ②. Rules, Jeff's input and every engine decision are unchanged. All payload and behaviour fingerprints are byte for byte the same, and research replay rebuilds every request.
1. Environment interface, env-v1 (c514e72)
environment.pyputs a line between the game and whoever plays a chef. An agent gets anAgentPort, never the kitchen object.observe()/act(): plain data, the contract any agent can rely on.observe()gives the kitchen in English without display-only fields, every listed action unpruned ({key, kind, target, label}), and the event sequence.act()is checked when the action arrives, with an optionalexpect_jobguard.view: the engine read-only, for in-process agents. Every engine input (any method marked byengine_input) is refused, and so is attribute assignment. Whatever an agent does therefore goes through the port, is recorded as an engine input, and replays exactly.jev.DecisionLoopreads through the view and starts actions and sprints through the port. Its 59 direct kitchen accesses are now read-only, and its 4 writes go through the port.agent_loop.AgentLoopruns anyenvironment.Agentin the game's session when the client factory returns one.example_agent.PriorityAgentis a complete agent built onobserve()/act()alone and imports nothing of Jeff's. It is a floor to beat, not a cook.environment.pyandagent_loop.pyship inRUNTIME_FILES. A new test checks that every project module a runtime file imports is in that list; a mutation check confirms it names the missing module.docs/architecture/environment-api.md(English, then Chinese).2. Benchmark v1 (e62e6e8)
The decisions approved in review: v1 covers levels 1–3 with the
assemblerpartner, and level 4 comes in v1.1; Jeff's rule prose stays on his side; results live in the repo as JSON plus a generated table.benchmark/v1/manifest.jsonfreezes what v1 measures:env-v1;reference_policy.py;A test fails when the code no longer matches the manifest.
scripts/benchmark.py run --agent jeff|module:Class.result.jsonreports money and goal rate with 95% intervals, the gain over the scripted chef alone, and the agent's share of the work.--dry-runprices a run: 36 rounds, 108 minutes of play.verifyrefuses a result that was:reportputs results side by side.Docs.
benchmark/README.md(English, then Chinese), plus pointers from both READMEs anddocs/agent-integration.md.3.
decision_statsintervals, decision-stats-18 (0a5bc24)This builds on 【云端】's decision-stats-17. A new
uncertaintysection gives 95% intervals for:Every other field is unchanged. Running v17 and v18 on the same journals gives identical reports apart from
uncertaintyand the version, with and without--by-roundand--split.Verification (0a5bc24)
python3 -W always::ResourceWarning -m unittest discover -s tests: 1180 tests pass, with 0 unclosed-database warnings and nologs/afterwards.node tests/*.cjsbrowser tests pass.scripts/audit_release.py: no findings.verifyin 181 s.--dry-run.Generated by Claude Code