Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Open work lives in a Product Grid project in `pgm/data` (project `srcly-backlog`

## Engine

`engine/` is the data-flow engine: tsx-data-flow's analysis behind a stdio protocol, a standalone pnpm package with its own lockfile. Provenance is in `engine/UPSTREAM.md`; the plan is `docs/intent/tsx-dataflow-integration/detailed-plan.md`.
`engine/` is the data-flow engine: tsx-data-flow's analysis behind a stdio protocol, a standalone pnpm package with its own lockfile. Provenance is in `engine/UPSTREAM.md`; the design rules and decisions are in `docs/dataflow-engine.md`.

- Install and check: `cd engine && pnpm install && pnpm test && pnpm typecheck && pnpm lint`.
- Run it from the server in a checkout with `SRCLY_ENGINE_DEV=1` (`dev.sh` and `scripts/dev-server.sh api` set it). The server runs `engine/bin/srcly-engine.ts` through the engine's own `tsx`, ahead of any bundle.
Expand All @@ -65,7 +65,7 @@ Open work lives in a Product Grid project in `pgm/data` (project `srcly-backlog`
- After merging, `survey-main` diffs each push to `main` against its first parent and compares that with the merged PR's own survey, so two PRs that move traces only together surface. It fails and keeps one issue, `Data-flow gate: unacknowledged moves on main`, up to date, and closes it on the next clean push.
- Open a data-flow PR, then run `gh pr merge <n> --squash --auto` and stop: auto-merge lands it when the checks pass. Don't run your own multi-repo surveys to prove a rule unless it targets code the gate's corpus doesn't contain.
- A new relation, element or proof kind gets one entry in `engine/src/srcly/kinds.ts` (`RELATION_TRAITS`, `ELEMENT_TRAITS`, `PROOF_TRAITS`); the engine does not build until it has one. Rules read its traits (`structural`, `crossCall`, `terminal`, a proof's `strength`), never their own lists of kind names.
- Never edit `engine/src/analysis` to recognise srcly-specific code. The only planned analysis edits are M5's four invocation-identity changes, each with a negative fixture.
- Never edit `engine/src/analysis` to recognise srcly-specific code. An analysis edit that a rule needs comes with a negative fixture.

## Client Package Management

Expand Down
2 changes: 1 addition & 1 deletion client/src/dataflow/contract.generated.ts
Original file line number Diff line number Diff line change
Expand Up @@ -613,7 +613,7 @@ export type FileSignalsResult = {
};
}[];
findings: {
rule: "impossible-defense" | "repeated-fork";
rule: "impossible-defense" | "repeated-fork" | "context-relay" | "broad-prop-bundle";
file: string;
line: number;
column: number;
Expand Down
4 changes: 2 additions & 2 deletions docs/CODE_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -420,7 +420,7 @@ Graph legibility rules (keep these when you change the layout or canvas):

### Data flow (engine)

The tsx-data-flow analysis lives in `engine/`, a standalone Node package with its own lockfile. Plan: `docs/intent/tsx-dataflow-integration/detailed-plan.md`.
The tsx-data-flow analysis lives in `engine/`, a standalone Node package with its own lockfile. Design rules and decisions: `docs/dataflow-engine.md`.

| Feature | Where |
| --- | --- |
Expand All @@ -446,7 +446,7 @@ The tsx-data-flow analysis lives in `engine/`, a standalone Node package with it
| `/api/dataflow/*` endpoints | `server/app/routers/dataflow.py` |
| Engine `functionEffects` (M6, T14): a walk over the callable's body names each effect as a fact with a witness (families external, input, write, throw, timer, promise; activation direct, after-await, timer, latent), follows first-party callees the checker resolves (memoized, bounded, cycle-safe, aliases followed) and merges their facts behind a `via` chain, names what it cannot follow as frontiers (callback-parameter, dynamic-dispatch, unresolved-callee, recursion, budget), and says how each promise-yielding call is handled; identity from the checker, never a name; TypeScript lib intrinsics taken as pure unless named | `engine/src/analysis/effects/function-effects.ts`, `engine/src/srcly/ops/function-effects.ts`, contract `engine/src/srcly/contracts/function-effects.ts`; fixture `engine/examples/effects-basics/src/cases.ts` (21 cases) and `engine/test/srcly/function-effects.test.ts`; Python `models.py:FunctionEffectsResult` |
| Explain an edge (M4, T16): `routePath {route, source?, edge}` maps the field attachment's, frontier's or handoff bridge's `evidencePathElementIds` through the slice into ledger rows (input, transfer, transform, branch, terminal, boundary at a frontier), never inventing a step; the client selects a handoff or field edge by click and the inspector lists its steps, each opening its line | `engine/src/srcly/ops/route-path.ts`, contract `engine/src/srcly/contracts/route-path.ts`, `engine/test/srcly/route-path.test.ts`; client `DependencyCanvas.tsx` (`onSelectEdge`), `FlowView.tsx` (`selectedEdge`, `edgePath`), `FlowInspector.tsx` (`EdgeStepsView`) |
| Data-flow signals (M7): tier-2 `fileSignals` op builds the legacy render-path report once and returns per-file metrics (impossible defenses, relay hops, fan-out, shape changes, path depth), `impossible-defense` and `repeated-fork` finding spans (a defense is placed in the file on its sink's path whose text matches) and `dataflow.md` (the old CLI's work-packets, defensive-ledger, repeated-forks, prop-relay, context-relay and fan-out views, 8 items each); `srcly report --dataflow` and `srcly compare --dataflow` run it; the last result per root is saved, and the app lays the six `dataflow_*` metrics over the tree and marks the code viewer from it while the fingerprint (tree) or file mtime (marks) still match | `engine/src/srcly/ops/file-signals.ts`, `engine/src/srcly/contracts/file-signals.ts`; `server/app/services/dataflow/signals.py` (`collect`, `apply_to_tree`, `markers_for`, `latest_for_file`); `reporting.py` (`scan_tree(dataflow=)`, `render_dataflow_section`); `compare/snapshots.py` (`with_dataflow`); overlay in `scan_jobs.py:_overlay`, marks in `routers/files.py:_dataflow_markers`; catalog group `Data flow` in `metric_catalog.py` |
| Data-flow signals: tier-2 `fileSignals` op builds the legacy render-path report once and returns per-file metrics (impossible defenses, relay hops, fan-out, shape changes, path depth), `impossible-defense`, `repeated-fork`, `context-relay` and `broad-prop-bundle` finding spans (relay rules from the report's context-relay pass; a defense is placed in the file on its sink's path whose text matches) and `dataflow.md` (the old CLI's work-packets, defensive-ledger, repeated-forks, prop-relay, context-relay and fan-out views, 8 items each); `srcly report --dataflow` and `srcly compare --dataflow` run it; the last result per root is saved, and the app lays the six `dataflow_*` metrics over the tree and marks the code viewer from it while the fingerprint (tree) or file mtime (marks) still match | `engine/src/srcly/ops/file-signals.ts`, `engine/src/srcly/contracts/file-signals.ts`; `server/app/services/dataflow/signals.py` (`collect`, `apply_to_tree`, `markers_for`, `latest_for_file`); `reporting.py` (`scan_tree(dataflow=)`, `render_dataflow_section`); `compare/snapshots.py` (`with_dataflow`); overlay in `scan_jobs.py:_overlay`, marks in `routers/files.py:_dataflow_markers`; catalog group `Data flow` in `metric_catalog.py` |
| Effects panel (M6, T15): Known facts by family and the places the analysis could not follow together, activation and via chain per fact, how each promise is handled, assumptions one click away, a witness row opens its line; in the code viewer's rail under the findings for the selected scope (else the function enclosing the opened line) and in the data-flow inspector for a selected box's function in anchor mode | `client/src/dataflow/EffectsPanel.tsx` (`createEffectsQuery`), pure wording in `client/src/dataflow/effectsModel.ts` (tested on the fake engine's shapes); hosts `CodeModal.tsx` → `MetricsSidebar.tsx` (`effects` slot) and `viz/deps/flow/FlowInspector.tsx` |
| Call identity (T13): the slice walk's invocation stack pairs a callee's return with the call the walk came in by (`pairInvocation`); a first-party callee is walked through its binding, never the bare `argument` link; the ops trust that pairing: a walk leaves a callee only through the call it entered by, and one that started inside it leaves through every call; an `argument` stays at its own call, and only the cross-calls the slice does not pair (callbacks, accessors) keep R7's one-call-site boundary (`facts.ts:needsOneCallSite`, `pairedInSlice`) | `engine/src/analysis/evidence-slice.ts`, `engine/src/srcly/facts.ts`, `engine/src/srcly/ops/{anchor-graph,value-flow}.ts`; fixtures `examples/value-flow-basics/src/{two-call-identity-helper,returns-other-argument}.tsx` |
| Field paths: the slice walk carries which field of a value it is after (`fields`, k-limited like the invocation stack). Backward a field read pushes its field and an object pack is entered only through that field (or a spread); forward a packed field pushes it and only a read of that field pops it. A reference, binding or return keeps the path; any other step widens it to the whole value, as does an element reached again with another path. An object literal's `fields` attribute names each property's field by its start, where its `pack-field` relation points; a store path setter's literal keys (`setStore("left", "title", v)`) pack its value field by field into the state, and past a computed key or a predicate the value is the whole field there | `engine/src/analysis/evidence-slice-fields.ts` (`stepFieldPath`), `evidence-slice.ts`, `program-evidence-collector-expression-support.ts` (`objectPackAttributes`); fixture `examples/value-flow-basics/src/field-paths.tsx` |
Expand Down
1 change: 1 addition & 0 deletions docs/dataflow-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ These came from the Python data-flow usability audit (`docs/research/python-data

## Engine structure

- Design rules, decisions, measurements and the batch evidence behind reports and the treemap (`fileSignals`): `docs/dataflow-engine.md`.
- `flowSignature` reduces each function of a file to line-free keys (origins, ends, origin→end pairs, parameter reads, effects, stops, status). `srcly compare --dataflow` diffs them as "Data-flow changes". Python has no parameter reads or effects (not measured), and a caller of a changed function is not re-traced.

- Every evidence kind is classified once in `engine/src/srcly/kinds.ts` (#200). Rules read traits, not lists of kind names.
Expand Down
91 changes: 91 additions & 0 deletions docs/dataflow-engine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Data-flow engine: design

srcly's data-flow features come from one engine: the tsx-data-flow analysis, moved into `engine/` and run as one lazy Node child per scan root. This page keeps the design rules and the decisions behind them. For where each file lives, read the "Data flow (engine)" section of `docs/CODE_MAP.md`. For what each rule covers, read `docs/dataflow-coverage.md`. For commands, read the "Engine" section of `AGENTS.md`.

## Shape

```text
engine/ Node 18+, TypeScript, Zod (the analysis, moved intact)
bin/srcly-engine.ts stdio JSON lines: {id, op, params} -> {id, ok, result | error}, progress frames
src/srcly/session.ts tier 1 session; tier 2 report on demand
src/srcly/ops/*.ts one file per op
src/srcly/contracts/*.ts Zod schemas, the source of truth for every payload
server/app/services/dataflow/ Python owns the child: spawn, reply matching, timeouts, kill
server/app/routers/dataflow.py /api/dataflow/*
client/src/dataflow/ generated contract types, API calls, outcome labels
```

Three surfaces read the engine: the route graph in the dependency view, the value ledger from the code viewer, and the effects panel. srcly's batch pipelines (metrics, findings, marks, reports, compare) read it through `fileSignals` and `flowSignature`.

## Invariants

1. **The base scan never waits.** The engine starts on first explicit use: Data flow mode, Inspect value flow, the effects panel, or a `--dataflow` CLI flag. The treemap, explorer and import graph are the same with or without Node.
2. **Evidence is bound to a generation.** Every result carries `generation` and `programHash`. A query with an older `expectedGeneration` gets `stale`, and the client does not draw stale rows over current source.
3. **Not analyzed is never 0.** Each surface shows the query outcome by name. A metric the engine did not measure is `None` with an `na_reason`.

## Session tiers

The legacy entry point built the full legacy report before it answered anything: 130 s on srcly's `client/`. So the session has two tiers that share one Program.

- **Tier 1** builds the primary Program and a lazy evidence provider, and finds routes. It answers `routes`, `routeGraph`, `routePath`, `selectExpression`, `valueFlow`, `functionEffects`, `flowSignature` and the anchor ops. `routes` never calls `buildRouteDataInventory`, which alone costs 17.5 s on visual-notes.
- **Tier 2** is the legacy report (`fullReport()`), built only by `fileSignals`. It is slow, so only a batch caller asks for it: `srcly report --dataflow` and `srcly compare --dataflow`.
- `programHash` hashes the sorted `(file, size, mtime)` of the Program's source files, every governing tsconfig, the TypeScript version and the engine version. `status` recomputes it with `stat` calls only. A rebuild bumps `generation`.
- Refresh drops both tiers. There is no incremental invalidation.

## Outcomes and positions

Each reply has one outcome: `ok`, `partial` (stops at named frontiers), `not-analyzed` (with a reason: no Node, no config, crash, timeout, memory limit), `unsupported`, `stale`, `cancelled` or `budget` (with counts). Each node, edge and ledger row has an evidence status: `proven`, `partial` or `unsupported`, with the engine's typed reason. Nothing turns these into a score or a percentage.

All contracts use 1-based lines, 1-based UTF-16 columns and exclusive end columns. Python and the client never convert them. Tree-sitter byte offsets (srcly's own markers) are never mixed with engine spans.

## One contract, three languages

The Zod schemas in `engine/src/srcly/contracts/` are the source of truth. `pnpm emit-contracts` writes JSON Schema for Python and `client/src/dataflow/contract.generated.ts`. The Pydantic models are hand-written, and a test checks their fields against the schema. `contracts-current.test.ts` fails when the emitted files drift. Python refuses a child whose protocol version differs; there is no coexistence layer.

## Containment

- The child reads only files that the Program owns under the root. Python checks that every `file` parameter resolves under the scan root before it reaches the child.
- The child runs in its own process group and is killed as a group. Its argv is fixed: `node`, the entry path and fixed flags. There is no shell.
- The engine never runs target code. It only parses and type-checks it.

## Batch evidence (`fileSignals`)

`fileSignals` builds the tier 2 report once and returns:

- per TS/TSX file: impossible defenses, relay hops, fan-out, shape changes and path depth. Python adds repeated forks from the findings. These are the six `dataflow_*` catalog metrics (`server/app/services/dataflow/signals.py`), overlay metrics like findings and activity.
- findings, one per site:
- `impossible-defense`: a `??` or `?.` whose guarded type is never null or undefined;
- `repeated-fork`: a component that tests one discriminant at several sibling branch sites;
- `context-relay`: a parent that reads a context hook and passes context-like props (`settings`, `selection`, `on*`, …) to a child in the same feature;
- `broad-prop-bundle`: the same, with three or more plain props.
- `dataflow.md`: the legacy CLI's work packets, defensive ledger, repeated forks, prop relay, context relay and fan-out views.

The last result per root is saved on disk. The app colors the treemap and marks code-viewer lines from it while the file is unchanged. The app never starts tier 2 itself.

## Decisions

| Decision | Choice | What would change it |
| --- | --- | --- |
| TypeScript instance | The bundled TypeScript; the project's copy only when its major.minor matches | A real project that does not parse with the bundled version |
| Tier 2 | Opt-in through `--dataflow` only | Tier 2 under 10 s on a 50K-line app |
| Process | Resident child, closed after 15 idle minutes, at most two alive | Tier 1 under 2 s (per-query spawn), or RSS above 3 GB (a cap of one) |
| Back and Return | A view-state registry (`client/src/navigation/viewState.ts`): snapshots in `sessionStorage` under a token in `history.state`, so a copied URL opens plain | Nothing expected |
| Progress | The client polls `status` every 250 ms | Polling that shows up in profiles (then an SSE endpoint) |

These were dropped from the plan:

- `dataflow_defensive_ops`: the legacy count is per sink, and one guard feeds many sinks, so a sum per file counts a guard many times. `dataflow.md`'s defensive ledger lists the guards.
- A `findings/dataflow.py` tool: data-flow findings go to `report.md` and code-viewer marks from the saved `fileSignals` result. They are not in the file-findings panel.

## Measurements

Development Mac, Node 22, TypeScript 5.9.3, one cold run each (2026-10-23 and 2026-10-24).

| Target | Tier 1 | Full legacy report | RSS |
| --- | ---: | ---: | ---: |
| srcly `client/` (26K lines, 126 files) | 3.0 s | 130 s | 2.4 GB |
| visual-notes `app/` (65K lines, 487 files, 43 routes) | 13.9 s | 44 s | 2.0 GB |

- Cold `hello` is about 0.6 s. A warm `valueFlow` on srcly's client is 1 ms at p50 and 3 ms at p95.
- After the backbone-first work, the first drawn route box on visual-notes `/tasks` takes about 11 s cold and 3.4 s on reopen from the result cache.
- Memory is the binding limit for a resident child. `SRCLY_DATAFLOW_MEMORY_MB` and `pnpm memory-probe` (see `AGENTS.md`) keep it measured.
20 changes: 20 additions & 0 deletions docs/initiative-history.json
Original file line number Diff line number Diff line change
Expand Up @@ -213,5 +213,25 @@
],
"revision": "0a1c1133",
"follow_ups": []
},
{
"title": "Bring tsx-data-flow into srcly",
"slug": "tsx-dataflow-integration",
"outcome": "completed",
"retired_at": "2026-11-21",
"summary": "The tsx-data-flow analysis lives in engine/ as one lazy Node child per scan root. It feeds the route graph, the value ledger with Return, the effects panel, data-flow metrics, findings and code-viewer marks, srcly report --dataflow (dataflow.md work packets) and compare --dataflow, and it ships bundled in the wheel. The strict-tsconfig refusal became nullProof=false. Reconciled against origin/main 4727bd56 plus the retirement PR's context-relay and broad-prop-bundle findings.",
"reason": "All eight milestones shipped. M7 closed on 2026-11-21 (#398, #399, #400, #406 and the retirement PR). Byron approved retirement.",
"documentation": [
"docs/dataflow-engine.md",
"docs/dataflow-coverage.md",
"docs/CODE_MAP.md",
"AGENTS.md",
"skills/srcly-code-quality/SKILL.md"
],
"revision": "4727bd56",
"follow_ups": [
"pgm task-start-the-data-flow-batch-run-from-the-app: start the tier-2 run from the app. Today only the CLI starts it.",
"Dropped: dataflow_defensive_ops (a per-sink count double-counts shared guards) and a findings/dataflow.py tool (findings reach report.md and marks from the saved result)."
]
}
]
2 changes: 1 addition & 1 deletion docs/intent/dataflow-robustness/decision-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ last_updated: "2026-10-24"

# Decision log

Each decision stays open only to the evidence named in its last column. Decisions R1–R12 came from three rounds of questions to the maintainer on 2026-10-24. The rest are planning defaults. The earlier initiative's D1–D8 (`../tsx-dataflow-integration/detailed-plan.md` §8) still stand unless one is superseded here.
Each decision stays open only to the evidence named in its last column. Decisions R1–R12 came from three rounds of questions to the maintainer on 2026-10-24. The rest are planning defaults. The earlier initiative's D1–D8 (`https://github.com/byronwall/srcly/blob/4727bd56/docs/intent/tsx-dataflow-integration/detailed-plan.md` §8) still stand unless one is superseded here.

| Id | Decision | Accepted | Evidence that would reopen it |
| --- | --- | --- | --- |
Expand Down
Loading
Loading