Isolate and type fan-out payloads, check input at the front door, trace a Ctrl-C'd node - #76
Merged
Conversation
…ce a Ctrl-C'd node
Three defects in `grapharc/runtime/graph.py`, all of the same family: a
contract the module documents, enforced everywhere except at one boundary.
**Fan-out handed every worker the same object, and never typed it.** `_enter`'s
comment states the rule — "Nodes get a deep copy: the returned dict is the
*only* write channel" — but the copy was guarded by
`isinstance(state, BaseModel)`, and a `Send` payload can be anything. Two Sends
built from one dict therefore gave both parallel workers the same live dict:
workers reported: ['dict hits=1 id=139338015848512',
'dict hits=2 id=139338015848512'] # same id
shared dict after run: {'who': 'orig', 'hits': [1, 1]}
That is a data race between nodes that are supposed to be isolated, and the
mutations travel through a channel no node declared a write to and no trace
event records — so the write-permission model and the audit trail both miss it,
and which worker ran first decides the answer. Fan-out is the one place the
isolation matters most; it was the one place that skipped it.
The other half of the same boundary: `_check_goto_target` validated `Send.node`
and left `Send.arg` alone, so `input_schema` — whose docstring says it "types a
fan-out worker's Send payload" — checked nothing. A dict where a model was
declared reached the worker and failed as a bare `AttributeError` several frames
from the dispatcher that produced it; a wrong model class sharing a field name
did not fail at all. That `Send.node` is checked and `Send.arg` is not reads as
an oversight rather than a decision, so `Send.arg` now gets the same treatment,
with `StateTypeError` naming the node, the schema and what arrived.
Every payload is deep-copied now, whatever its type. Declaring no `input_schema`
stays legal — a worker that names no schema is claiming nothing, so there is
nothing to check — but the copy is unconditional, and `add_node`'s docstring now
says so instead of leaving it to be inferred. A `BaseModel` payload was already
isolated correctly and still is, asserted by its own test.
**The front door was the one door the state contract did not hold.**
`state.py` opens by promising "a typo'd update key fails loudly at the edge
instead of silently polluting downstream nodes". `update_state` keeps that
promise and so does constructing the model; the entry points did not. They
handed `input` to LangGraph, which filters a dict down to known channels
*before* the state model is ever constructed, so `extra="forbid"` never got a
chance:
invoke({'quesiton': 'typo'}) -> {'out': "saw:''"} # ran on defaults
update_state({'quesiton': 'typo'}) -> WritePermissionError: unknown state fields
Misspell the field carrying the question and you get a complete, plausible run
against an empty question and are told nothing — the quietest failure in the
runtime, on the door every user goes through first. `invoke`, `stream`,
`ainvoke`, `astream` and `astream_events` now refuse an unknown input key, in
the same words `update_state` uses, from the same helper so the two cannot
drift. A wrongly *typed* input value was already loud and still raises
Pydantic's `ValidationError`; a state model or `None` is passed through
untouched.
**A Ctrl-C'd sync node left no ending in the trace.** The sync wrapper caught
`Exception`; the async twin catches `BaseException`, and its comment gives the
reason — "a stop with no trace line is a stop nobody can audit afterwards". So
the same node body, run each way:
sync trace phases: [topology, start]
async trace phases: [topology, start, error]
`metrics.summarize` then reported `errors: 0` for the sync run, and an audit
reads it as having simply stopped between nodes. Ctrl-C is not an exotic
ending; it is the commonest way a human stops a long run. The sync wrapper now
matches its twin. The exception is re-raised untouched — only the trace write
is new.
Tests: twelve, each failing on main. Fan-out — two workers sharing one dict
cannot see each other's mutations, a payload contradicting `input_schema` is
refused at dispatch, a wrong model class likewise, an untyped payload stays
legal, and a `BaseModel` payload is still isolated. Front door — `invoke` and
`stream` reject the typo verbatim, `ainvoke`/`astream` in a parametrised twin,
with the type-error and valid-input paths pinned unchanged. Ctrl-C — one test
asserts both wrappers write the `error` event for `KeyboardInterrupt` and
`SystemExit`, so the two paths cannot drift again.
Deep-copying every payload costs nothing measurable: 200 fan-out workers run in
41.1ms with the change and 41.6ms without (best of five, dict payloads, well
inside run-to-run noise), because 200 `copy.deepcopy` calls on a realistic
shard total 0.5ms against ~41ms of graph execution.
Fixes #67
Fixes #68
Fixes #70
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…and-input-validation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three defects in
grapharc/runtime/graph.py, all of the same family: a contract the module documents, enforced everywhere except at one boundary. They share a file, so they share a PR.#67 — fan-out payloads bypassed isolation and were never typed
_enter's own comment states the rule: "Nodes get a deep copy: the returned dict is the only write channel." The copy was guarded byisinstance(state, BaseModel), and aSendpayload can be anything. Two Sends built from one dict therefore handed both parallel workers the same live object:That is a data race between nodes that are meant to be isolated, and the mutations travel through a channel no node declared a write to and no trace event records — so the write-permission model and the audit trail both miss it, and which worker got there first decides the answer. Fan-out is where the isolation matters most; it was the one path that skipped it.
The other half of the same boundary:
_check_goto_targetvalidatedSend.nodeand leftSend.argalone, soinput_schema— whose docstring says it "types a fan-out worker's Send payload" — enforced nothing.Every payload is deep-copied now, whatever its type, and
Send.arggets the treatmentSend.nodealready got. Declaring noinput_schemastays legal — a worker that names no schema is claiming nothing — but the copy is unconditional, andadd_node's docstring now says so rather than leaving it to be inferred. ABaseModelpayload was already isolated correctly and still is, with its own regression test.#68 — the front door was the one door the state contract did not hold
state.pyopens by promising "a typo'd update key fails loudly at the edge instead of silently polluting downstream nodes."update_statekept that promise; the entry points handedinputto LangGraph, which filters a dict down to known channels before the state model is ever constructed, soextra="forbid"never got a chance.Misspell the field carrying the question and you got a complete, plausible-looking run against an empty question, with nothing said to the caller — the quietest failure in the runtime, on the door every user goes through first. All four entry points plus
astream_eventsnow refuse an unknown key, from the same helperupdate_stateuses so the wording cannot drift. Unchanged on purpose: a wrongly typed input value still raises Pydantic'sValidationError(invoke({"out": 5})), and a state model orNonepasses through untouched.#70 — a Ctrl-C'd sync node left no ending in the trace
The sync wrapper caught
Exception; the async twin catchesBaseException, and its comment gives the reason — "a stop with no trace line is a stop nobody can audit afterwards."metrics.summarizethen reportederrors: 0for the sync run, and an audit reads it as having simply stopped between nodes. Ctrl-C is not an exotic ending; it is the commonest way a human stops a long run. The wrapper now matches its twin and re-raises the exception untouched — only the trace write is new.Tests
Twelve, each verified failing before its fix:
tests/test_runtime_discipline.py) — two workers sharing one dict cannot observe each other's mutations; a payload contradictinginput_schemais refused at dispatch; a wrong model class likewise; an untyped payload stays legal; aBaseModelpayload is still isolated (regression guard, passes before and after).tests/test_runtime_discipline.py,tests/test_async_kernel.py) —invokeandstreamreject the typo verbatim,ainvoke/astreamin an async twin; the type-error and valid-input paths are pinned unchanged.tests/test_async_kernel.py) — one parametrised test asserts both wrappers write anerrorevent forKeyboardInterruptandSystemExit, so the two paths cannot drift again. Driven withasyncio.runrather than@pytest.mark.asyncio, because asyncio re-raises these two out of the task step and into the loop.Scope
Deliberately untouched: budget metering, deadline guards, write permissions, and trace event shapes beyond the one missing
errorevent.Performance
Deep-copying every payload is free at graph scale — 200 fan-out workers, dict payloads, best of five:
Run-to-run noise dominates. In isolation, 200
copy.deepcopycalls on a realistic shard cost 0.5 ms and 200model_copy(deep=True)cost 0.7 ms, against ~41 ms of graph execution.Verification
ruff check .clean. Full suite: 1811 passed, 1 failed. The single failure istest_cookbook_agents.py::test_snippet_runs_as_printed[11-sandbox-refusals.py], which is environmental and fails identically on untouchedmainin this worktree — the snippet normalises a<site-packages>path and the worktree's symlinked.venvresolves elsewhere. None of the SIGALRM timing tests flaked.Fixes #67
Fixes #68
Fixes #70
🤖 Generated with Claude Code