Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Open work lives in a Product Grid project in `pgm/data` (project `srcly-backlog`
- To add an op: a Zod contract in `engine/src/srcly/contracts/` (and its `registry.ts` entry) → the op in `engine/src/srcly/ops/` and its worker branch → `cd engine && pnpm emit-contracts` → a Pydantic mirror in `server/app/services/dataflow/models.py` (`tests/test_dataflow_models.py` checks its fields against the schema) → a route in `server/app/routers/dataflow.py` → the client call. The emitted `server/app/services/dataflow/contracts/*.schema.json` and `client/src/dataflow/contract.generated.ts` are never edited by hand; `engine/test/srcly/contracts-current.test.ts` fails when they drift. Then `cd server && SRCLY_UPDATE_SNAPSHOTS=1 uv run pytest tests/test_dataflow_outcomes.py` writes the `tests/snapshots/fake-engine.*.json` files that `client/src/dataflow/shapes.test.ts` reads.
- Every result carries `generation` and `programHash`. A query with an older `expectedGeneration` is `stale`, a named outcome; never mix generations.
- Not analyzed is never 0: show the outcome or reason by name.
- Python roots: `engine/python/srcly_pyfacts.py` (stdlib `ast` + mypy, run with srcly's own interpreter) emits facts in the collector's raw shape; roles and boundaries come from the element kind, so new Python recognition goes in the helper's tables, never in `engine/src/analysis`. Its tests need an interpreter with mypy (`SRCLY_PYTHON`, else `server/.venv` after `cd server && uv sync`). Decision: `docs/intent/python-dataflow/decision.md`.
- Python roots: `engine/python/srcly_pyfacts.py` (stdlib `ast` + mypy, run with srcly's own interpreter) emits facts in the collector's raw shape; roles and boundaries come from the element kind, so new Python recognition goes in the helper's tables, never in `engine/src/analysis`. Its tests need an interpreter with mypy (`SRCLY_PYTHON`, else `server/.venv` after `cd server && uv sync`). Why `ast` + mypy: `docs/dataflow-coverage.md` (Python type source).
- Coverage survey: `engine/scripts/survey-blockers.ts` traces about 120 exports of one repo both ways and writes every outcome and stop as JSON; `engine/scripts/survey-report.ts <dir>` ranks the stops by how many traces each cuts short.
- Any script that builds a Program on a real repo (the survey, `pnpm memory-probe`, an ad-hoc spike) runs under `node --max-old-space-size=4096`, one repo per process, one process at a time across all sessions: a big repo's Program takes 2–3 GB, and parallel runs swap the Mac. `~/Projects/visual-notes` is the standard big-repo benchmark.
- A packaging change is done when the wheel works with dev mode off: `bash build-srcly.sh`, then `uv run --no-project --with ./server/dist/srcly-*.whl pytest scripts/test_package.py` (CI runs the same).
Expand Down
2 changes: 1 addition & 1 deletion docs/CODE_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -453,7 +453,7 @@ The tsx-data-flow analysis lives in `engine/`, a standalone Node package with it
| Fixture projects (Solid route, compact field proof, bad-ish Solid, Python FastAPI) | `engine/examples/` |
| Python handler routes in Data flow mode: the route summary's `handler` (name, line) opens the handler in anchor mode from the picker or a `flow=` link; Python name reads are `transparent` (walked through, not drawn) | engine `contracts/routes.ts`, `ops/routes.ts`, `ops/anchor-graph.ts` (`walk`); client `viz/deps/flow/FlowView.tsx` (`handlerOf`, `pickRoute`); helper `expr_Name` |
| Language host: what the language-neutral ops (`routes`, `selectExpression`, `valueFlow`, `anchorGraph`) ask a program besides its evidence (files, candidates, parts, identifier on a line, text, checked type, owning function, call sites, and `evidence` — whether its facts pair calls (T13) and mark real dead ends (R16)); `Tier1` is `TsTier1 \| PythonTier1`, `requireTypescript` guards the TypeScript-only ops (route graph and surface, route path, function effects, file exports, legacy report, phase profiler); engine flags `--language`, `--python` | `engine/src/srcly/host.ts`, `typescript-host.ts`, `python/host.ts`, `python/session.ts`, `session.ts`; tests `engine/test/srcly/python-session.test.ts`, server `tests/test_dataflow_availability.py`, `tests/test_dataflow_smoke.py` (FastAPI example and srcly's own server over HTTP) |
| Python evidence provider: `ast` + mypy → raw ProgramElement / ProgramRelation facts, selectable expressions with checked types, function and class scopes (a class is an entry its constructor calls relate to by `definition`), call sites, FastAPI / Flask routes (one JSON document per generation); newer syntax than the interpreter's (3.14's `except A, B:`) rewritten before parsing; loaded behind the fact index by an endpoint-map relation provider; refusals `python-not-found`, `python-checker-not-found`, `no-python-files` | `engine/python/srcly_pyfacts.py` (origins and terminals from the `py` rows of `engine/models/python.json`, passed with `--models`); `engine/src/srcly/python/run.ts` (spawn, probe, named errors), `facts.ts` (wire format, `PythonRelationProvider`); fixtures `engine/examples/python-fastapi/`, `engine/examples/python-anchors/` (every def and class anchored by name); tests `engine/test/srcly/python-facts.test.ts`, `python-anchors.test.ts`; decision and numbers `docs/intent/python-dataflow/decision.md` |
| Python evidence provider: `ast` + mypy → raw ProgramElement / ProgramRelation facts, selectable expressions with checked types, function and class scopes (a class is an entry its constructor calls relate to by `definition`), call sites, FastAPI / Flask routes (one JSON document per generation); newer syntax than the interpreter's (3.14's `except A, B:`) rewritten before parsing; loaded behind the fact index by an endpoint-map relation provider; refusals `python-not-found`, `python-checker-not-found`, `no-python-files` | `engine/python/srcly_pyfacts.py` (origins and terminals from the `py` rows of `engine/models/python.json`, passed with `--models`); `engine/src/srcly/python/run.ts` (spawn, probe, named errors), `facts.ts` (wire format, `PythonRelationProvider`); fixtures `engine/examples/python-fastapi/`, `engine/examples/python-anchors/` (every def and class anchored by name); tests `engine/test/srcly/python-facts.test.ts`, `python-anchors.test.ts`; decision and numbers in `docs/dataflow-coverage.md` (Python type source) |
| Field proof acceptance runner | `engine/scripts/accept-route-field-proof.ts` |

## Tests
Expand Down
15 changes: 15 additions & 0 deletions docs/dataflow-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,21 @@ The work came from the data-flow experience audit (October 2026). The audit trac
- State: a zustand store (a `store` row) holds what its function returns and what its `set` receives; a selector reads the field it names, and calling a member read through the hook (`setCount` after `useStore((s) => s.setCount)`) runs the store's function, never the selector it is written beside. a setter's value reaches its getter's reads (#182); a function handed to a setter (`setCount((c) => c + step)`, a store path's `setState("todos", (list) => […])`) stands for what it returns, and its parameter is the current state; a Solid store path (`setStore("left", "title", v)`) writes `left.title` alone, so a sibling field never sees it, an updater there (inline or named) receives that field, not the whole store, and a predicate in the path (`(item) => item.id === id`) is a test, never stored; a store or signal's initial value seeds it field by field, so `createStore({ left, right })` keeps `left` and `right` apart; a state-pair row marked `lazy` (React's `useState`) calls a function initial value, and Solid's `createSignal` holds it; a `useReducer` dispatch binds the reducer's action, and the reducer's return is the state; state-pair factory and getter calls are framework-modelled (#188), modelled in the collector (#196).
- Ends: a binding every read of which decides a branch (`if (forgotten)` in nested branches, `ready && <X />`, `<Show when={isVisible()}>`, through `!` or an accessor's call) ends at its binding, decides a branch. A parameter or local binding the program never reads (`_level`, `const { isSuccess } = useQuery()`) ends there, named "never read": the checker proves the value goes nowhere, so the trace is `ok`, not a dead end; the value-flow ledger draws every in-place end (a branch test, a write, never read) as a terminal row, as the graph does. A service write or read is a method on a client's contract, or a function-valued field a declaration file declares (a generated client's); a callback prop the project declares (`props.save(x)`) is never one. A thrown value ends at the throw (#207). An exported value that nothing in the program reads ends at its export, the module's API, not at a dead end; a call through a callback an export nothing in the program calls was handed (`onSeen(id)`, `props.onPick(x)` in a component nothing renders) ends there too, at the module's API, never a library method on a parameter (`items.forEach`); one the program reads, by name or through `import * as`, walks on to that read. A function handed to a timer (`setTimeout`, `requestAnimationFrame`) ends there: the callback runs later. A function handed to a page element as a handler (`<button onClick={click}>`) is not an escape: the page drops what it returns. A trace whose anchor has nothing beyond it names a stop at the anchor. A context `Provider`'s `value` goes to its readers, never into the page. A request ends a forward trace where it is sent, and a backward trace lands at the request a response came from: neither stops at the response body. A promise chain the code drops (`load().then(show);`) ends there. A function library code hands to a callback (`resolve` in `new Promise((resolve) => …)`) hands its argument back to the call that supplied it: the value reaches that call's value, and the call through it ends there, handed back. A DOM call the code makes for its effect (`el.classList.add(x);`, `target.dispatchEvent(e);`, `() => input?.focus()`) is a page write: any member TypeScript's DOM lib declares, whose value the code drops. A DOM call whose value the code uses (`root.querySelector(sel)`) carries on. An arrow whose body is an optional void call returns nothing. `response.json()` on a `fetch` result reads the body; only a server's response is an HTTP response. An emitter's `emit("name", x)` binds each listener its receiver registered with `on("name", fn)`, and `yield* list` hands on `list`'s elements. A return that is a call (`yield items.slice(…)`, an arrow's `() => f(x)`) is judged by its own function's call sites, never the call's, so an exported generator no code iterates ends at its return. A class's annotated uses fold by file (#208). A `when` on Solid's `Show` or `Match` ends as "decides a branch", not a DOM attribute.

## Python type source

A Python root uses the same engine contracts as TypeScript. `engine/python/srcly_pyfacts.py` provides the evidence. The engine runs it once per generation.

- Structure comes from the standard library's `ast`. Types and call targets come from mypy's `build.build` with `export_types`. The helper prints ProgramElement and ProgramRelation facts in the collector's raw shape.
- A box's type is mypy's type at that span (`typeSource: "mypy"`). Where mypy has none, such as a parameter or a file it could not check, the annotation text stands in (`typeSource: "annotation"`). A pydantic or dataclass field read cites the field's declaration as proof.
- mypy names the function or method a call reaches (`CallableType.definition`). This gives argument-to-parameter and return-to-call relations without guessing.
- mypy reads the project's own packages through `python_executable`: the project's `.venv`, else `VIRTUAL_ENV`, else the helper's interpreter. mypy is a dependency of srcly's server. When mypy is missing, the engine refuses with `python-checker-not-found`.
- Rejected: annotations over `ast` alone (would need a hand-written inference engine for unannotated code), `pyright --outputjson` (diagnostics only, no types) and pyright's language server (one hover at a time, about 65 s for `server/app`, display strings with no call structure).
- Measured on 2026-10-24 on srcly's `server/` (59 files): 4.9 s wall, of which mypy took 2.3 s, and 563 MB peak RSS. mypy typed 91% of expressions precisely.
- Origin and terminal roles come from each element's `kind`, so new Python recognition goes in the helper's tables and in the `py` rows of `engine/models/python.json`, never in `engine/src/analysis`.
- A file with syntax newer than the helper's interpreter is rewritten when an older spelling keeps every column: `except A, B:` (PEP 758) becomes `except(A, B):`. A file that still does not parse is a file gap.
- Name reads join their bindings flow-insensitively within a function. A read with more than one reaching binding is `partial`.
- A Python route opens its handler in anchor mode. `routeGraph` has no Python meaning.

## Python rules

- Any `def` or class anchors by name (#164). Calls pair by invocation (#166).
Expand Down
17 changes: 17 additions & 0 deletions docs/initiative-history.json
Original file line number Diff line number Diff line change
Expand Up @@ -167,5 +167,22 @@
"follow_ups": [
"Re-resolve name and typed edges with the data-flow engine's checker. The backlog area area-symbol-graph in pgm/data holds any open work."
]
},
{
"title": "Python data flow in the engine",
"slug": "python-dataflow",
"outcome": "completed",
"retired_at": "2026-11-12",
"summary": "Data flow works on Python roots: an ast + mypy evidence provider, a per-language session that names python-checker-not-found, and anchor flow from FastAPI routes. This folder held only decision.md and had no initiative map; a minimal one was written only to run the retirement check. Reconciled against origin/main 0bfb05ba.",
"reason": "All three planned slices merged on 2026-10-25 (#119, #120, #122), with many follow-on Python fixes after.",
"documentation": [
"docs/dataflow-coverage.md",
"AGENTS.md"
],
"revision": "0bfb05ba",
"follow_ups": [
"Not built: a model passed to a serializer (model_validate, jsonable_encoder) as a use of its class. Nothing tracks it.",
"Not built for Python: following values across methods through instance state (self.x = ...). Partly tracked as pgm task-writers-of-a-model-field."
]
}
]
Loading
Loading