Skip to content

Environment interface (env-v1), Benchmark v1, and intervals in decision_stats - #79

Merged
belchingleo merged 3 commits into
mainfrom
claude/compassionate-tesla-8bonjg
Nov 8, 2026
Merged

belchingleo merged 3 commits into
mainfrom
claude/compassionate-tesla-8bonjg

Conversation

@belchingleo

Copy link
Copy Markdown
Owner

Three commits on top of main (3ca48a0): review item ③ and the last part of ②. Rules, Jeff's input and every engine decision are unchanged. All payload and behaviour fingerprints are byte for byte the same, and research replay rebuilds every request.

1. Environment interface, env-v1 (c514e72)

environment.py puts a line between the game and whoever plays a chef. An agent gets an AgentPort, never the kitchen object.

  • observe() / act(): plain data, the contract any agent can rely on.
    • observe() gives the kitchen in English without display-only fields, every listed action unpruned ({key, kind, target, label}), and the event sequence.
    • act() is checked when the action arrives, with an optional expect_job guard.
  • view: the engine read-only, for in-process agents. Every engine input (any method marked by engine_input) is refused, and so is attribute assignment. Whatever an agent does therefore goes through the port, is recorded as an engine input, and replays exactly.
  • Jeff is the reference agent on this interface. jev.DecisionLoop reads through the view and starts actions and sprints through the port. Its 59 direct kitchen accesses are now read-only, and its 4 writes go through the port.
  • Any agent can play. agent_loop.AgentLoop runs any environment.Agent in the game's session when the client factory returns one.
  • Example. example_agent.PriorityAgent is a complete agent built on observe()/act() alone and imports nothing of Jeff's. It is a floor to beat, not a cook.
  • Packaging. environment.py and agent_loop.py ship in RUNTIME_FILES. A new test checks that every project module a runtime file imports is in that list; a mutation check confirms it names the missing module.
  • Docs. docs/architecture/environment-api.md (English, then Chinese).

2. Benchmark v1 (e62e6e8)

The decisions approved in review: v1 covers levels 1–3 with the assembler partner, and level 4 comes in v1.1; Jeff's rule prose stays on his side; results live in the repo as JSON plus a generated table.

  • benchmark/v1/manifest.json freezes what v1 measures:

    • env-v1;
    • each level's configuration hash and the hash of its golden reference trace, which changes exactly when the engine plays the level differently;
    • reference_policy.py;
    • 12 rounds per level, on a real clock.

    A test fails when the code no longer matches the manifest.

  • scripts/benchmark.py run --agent jeff|module:Class.

    • It plays every cell in real time, in separate processes, keeping each round's bundle and its replay check.
    • result.json reports money and goal rate with 95% intervals, the gain over the scripted chef alone, and the agent's share of the work.
    • --dry-run prices a run: 36 rounds, 108 minutes of play.
  • verify refuses a result that was:

    • made for another manifest;
    • made from a checkout that no longer matches it;
    • run with the wrong number of rounds;
    • played on a virtual clock;
    • containing a round that does not replay.
  • report puts results side by side.

  • Docs. benchmark/README.md (English, then Chinese), plus pointers from both READMEs and docs/agent-integration.md.

3. decision_stats intervals, decision-stats-18 (0a5bc24)

This builds on 【云端】's decision-stats-17. A new uncertainty section gives 95% intervals for:

  • the goal rate, the share of decisions that chose the top option, and the share of steps replaced by Jeff's own next choice (Wilson);
  • the mean money, cooperation gain and Jeff's standard-time share per round, and the median step gap per request policy (seeded bootstrap).

Every other field is unchanged. Running v17 and v18 on the same journals gives identical reports apart from uncertainty and the version, with and without --by-round and --split.

Verification (0a5bc24)

  • python3 -W always::ResourceWarning -m unittest discover -s tests: 1180 tests pass, with 0 unclosed-database warnings and no logs/ afterwards.
  • All 8 node tests/*.cjs browser tests pass.
  • scripts/audit_release.py: no findings.
  • A one-round real-clock benchmark run of the example agent went through the process-pool path and was accepted by verify in 181 s.
  • No real model was called. Paid benchmark runs are for the owner to run locally, after --dry-run.

Generated by Claude Code

Liyao Wang added 3 commits November 8, 2026 02:27
environment.py: an AgentPort per chef. observe() and act() are plain
dictionaries, the contract any agent can rely on: the kitchen state as the
engine has it in English without display-only fields, every listed action
unpruned, events since a sequence number, and act() checked when it arrives
with an optional job guard. view is the engine read-only: every engine input
(marked by the engine_input decorator) is refused, so all changes go through
the port and land in the round's recorded inputs. fork_until_step_end exposes
the isolated prediction ask_ahead uses.

Jeff's DecisionLoop reads the kitchen through the view and starts actions and
sprints through the port; payload and behaviour fingerprints are byte for byte
unchanged. agent_loop.AgentLoop runs any environment.Agent in the game's
session when the client factory returns one; example_agent.PriorityAgent
plays whole rounds on observe() and act() alone, imports nothing of Jeff's,
and its bundles replay exactly.

environment.py and agent_loop.py ship with the runtime, and a test now checks
that every project module a runtime file imports is in RUNTIME_FILES.
benchmark/v1/manifest.json freezes levels 1-3 (configuration hash and the hash
of each golden reference trace), the scripted assembler partner
(reference_policy.py), 12 rounds per level on a real clock and env-v1; a test
fails when the code no longer matches it. scripts/benchmark.py runs every cell
for Jeff on the saved connection or any environment.Agent (module:Class), in
real time and in separate processes, keeping each round's bundle and its replay
check; result.json reports money and goal rate with 95% intervals, the gain
over the scripted chef alone and the agent's share of the work. verify refuses
a result made for another manifest, from a checkout that no longer matches it,
with the wrong number of rounds, on a virtual clock or with a round that does
not replay; report puts results side by side; --dry-run prices a run.

Docs: docs/architecture/environment-api.md and benchmark/README.md (English,
then Chinese), pointers from both READMEs and docs/agent-integration.md,
changelog entries. model_rounds reports each round's configuration hash and
model.
A new uncertainty section gives 95% intervals for the goal rate, the share of
decisions that chose the top option and of steps replaced by Jeff's own next
choice (Wilson), and for the mean money, cooperation gain and Jeff's
standard-time share per round and the median step gap per request policy
(seeded bootstrap, research_stats.py). Every other field is unchanged; run
from the same place on the same journals, decision-stats-17 and -18 agree on
all of them, with and without --by-round and --split.
@belchingleo
belchingleo merged commit 0a5bc24 into main Nov 8, 2026
4 checks passed
@belchingleo
belchingleo deleted the claude/compassionate-tesla-8bonjg branch November 10, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant