Skip to content

refactor(pipeline): run the parallel path for every index - #118

Merged
ata-sesli merged 3 commits into
simplify/zova-ops-tablefrom
simplify/single-pipeline-path
Nov 3, 2026
Merged

ata-sesli merged 3 commits into
simplify/zova-ops-tablefrom
simplify/single-pipeline-path

Conversation

@ata-sesli

Copy link
Copy Markdown
Member

Summary

Implements #110. Every index, full or incremental, now goes through the parallel extract/resolve path, whatever its file count or worker count. The sequential passes are deleted, and the parallel resolver gained the results that only the sequential one used to produce.

Stacked on #117. This PR's base is simplify/zova-ops-table. Review and merge #116 and #117 first; GitHub then retargets this PR to main.

Problem

Indexing had two implementations:

  • Sequential. pass_definitions.c, pass_calls.c, pass_usages.c, pass_semantic.c and cbm_pipeline_pass_lsp_cross. It ran for repositories with 50 files or fewer, for CBM_WORKERS=1, and for incremental runs touching 50 files or fewer.
  • Parallel. pass_parallel.c. It ran for everything else.

Each resolver rule had to be written twice, and "keep in sync" comments asked for that by hand. The two copies had drifted. The same project could get different edges depending on how many files it had or how many workers ran it.

Change

Three commits.

1. fix(pipeline): give the parallel resolver what only the sequential one had

I forced the parallel path for every index and ran the full suite. These were the only failures, and this commit fixes them in pass_parallel.c:

  • Interface bases. A class base that resolves to an Interface becomes IMPLEMENTS, not INHERITS.
  • FastAPI Depends(fn). It becomes a CALLS edge (strategy: fastapi_depends).
  • Local fetch. A call to the project's own fetch is no longer turned into HTTP_CALLS by the arg-URL heuristic (#856).
  • Cross-file LSP gate. The gate counted confidence-0 lsp_unresolved rows, so files whose only call was unresolved skipped cross-file LSP. It also ignored callable-reference candidates. It now counts only the rows the CALLS join uses, and also runs when reference candidates outnumber reference rows.

2. refactor(pipeline): run the parallel path for every index

Pipeline routing:

  • Full index. pipeline.c always runs run_parallel_pipeline. run_sequential_pipeline and MIN_FILES_FOR_PARALLEL are gone.
  • Incremental index. pipeline_incremental.c always extracts and resolves changed files in parallel.
  • Callers of changed files. These are re-resolved through a new cbm_parallel_resolve_calls in pass_parallel.c, which uses the same resolve_file_calls as a full index.

Deleted:

  • pass_definitions.c and pass_usages.c.
  • cbm_pipeline_pass_lsp_cross.
  • The sequential half of pass_semantic.c. The Go implicit-interface check remains in that file.
  • cbm_pkgmap_build_from_repo / cbm_pkgmap_build_from_files.
  • The sequential result cache.

Moved and kept:

  • pass_calls.c is now service_bindings.c. It keeps only the two helpers the parallel resolver calls: cbm_pipeline_http_client_call_url (#1916) and cbm_pipeline_external_import_service.
  • pass_k8s.c always extracts kustomize and manifest files itself; it no longer reads them from the deleted cache.

CBM_WORKERS=0 already meant one worker. The comment and docs/CONFIGURATION.md no longer say it selects a sequential path.

Tests:

  • Parity tests in test_parallel.c. They compared the deleted passes against the parallel path. They now compare a 1-worker run against a 2-worker run.
  • Removed test. pipeline_calls_args_overlong_argument_stops_cleanly tested a formatter that only existed in pass_calls.c; parallel_args_json_no_overflow covers the parallel one.
  • Arg-URL control. The control in typed_http_no_url_twin_case was skipped for the small fixture because the sequential resolver lacked the heuristic. It now runs for both fixture sizes.

3. fix(pipeline): drop the self-loop of an unresolved route-suffix call

Small-repo parity found one more gap. An unresolved call whose name ends like a route method (request.headers.get('content-type'), unified().use(plugin)) with no route path goes through the callee_suffix fallback. That fallback passes the caller as the target, so its plain CALLS fall-through made a caller→caller self-loop. The sequential resolver never emitted it. Large repos on main already had these self-loops, because they took the parallel path.

New test: pipeline_unresolved_route_suffix_call_adds_no_self_loop. It fails without the fix.

Diff: 30 files, about 3,000 fewer lines.

Output compared with main

Measured with the CLI (index_repository into a fresh cache), main at 616926c vs this branch, macOS arm64. Counts are per node label and per edge type from get_graph_schema.

Repository Files Nodes Edges
medium-cli (TypeScript) 27 indexed identical (205) identical (511)
mini-observer (C, Python) 19 indexed identical (467) identical (1,411)
this repository (snapshot of main) 686 indexed identical (22,655) 153,511 → 153,534

The two small repositories took the sequential path on main. They give identical counts, both with the default worker count and with CBM_WORKERS=1.

The large-repository differences are expected:

  • CALLS: 36,889 → 36,887. These are the two self-loops removed by commit 3 (httpGet and fetchChecksums in the Go installer).
    • One more edge keeps its endpoints but changes strategy from same_module to lsp_direct, because the cross-file LSP gate in commit 1 now runs for that file.
  • SEMANTICALLY_RELATED: 1,940 → 1,965.
    • The changed pairs are almost all near-identical test functions (tests/test_c_lsp.c against other LSP test files) trading one nearest neighbour for another.
    • Semantic vectors are built partly from each function's CALLS neighbours, and CALLS is the only other edge type that changed. These flips most likely come from that change, but I did not isolate the cause with a separate build.
    • Both binaries are deterministic: three runs each gave identical counts.

The other commit 1 fixes add or retype edges only for code this repository does not have: Java/C#/TS interfaces, FastAPI, and a local fetch. On other projects, expect more IMPLEMENTS (fewer INHERITS), new fastapi_depends CALLS, more cross-file LSP CALLS / CALL_REFERENCE, and no HTTP_CALLS for calls to a project's own fetch.

Index time

Wall time of the cli index_repository process; 5 runs per small case, 3 for the large one; median shown.

Repository Workers main this branch
medium-cli default 0.474 s 0.368 s
medium-cli 1 0.582 s 0.464 s
mini-observer default 0.687 s 0.463 s
mini-observer 1 0.893 s 0.895 s
this repository default 17.40 s 17.85 s

Small repositories are as fast or faster on the parallel path. That includes one worker, which used to mean the sequential path.

The large repository is about 2.6% slower (0.45 s over 3 runs each). The likely cause is the wider cross-file LSP gate from commit 1. Peak RSS was 1.11–1.20 GB on main and 1.21–1.28 GB on this branch.

Verification

  • Full test suite with CBM_TEST_TIERS=all (macOS arm64): 6,449 passed, 0 failed.
    • That is the 6,449 baseline, minus the deleted pass_calls formatter test, plus the new self-loop test.
    • The probes' UNEXPECTED PASS notes are identical to the base's.
  • New test fails without its fix: pipeline_unresolved_route_suffix_call_adds_no_self_loop failed with ASSERT(!(self_loop)) before commit 3.
  • Build sources: scripts/check-build-sources.sh passes after the build-list changes.
  • Format and lint: clang-format-diff (clang-format 22.1.8) is clean on the changed lines; git diff --check and tests/test_github_workflows.sh pass.
    • Exception: service_bindings.c is the renamed pass_calls.c, and one signature carried over unchanged from main is wrapped differently from clang-format's output. I left it as on main.

Not done

  • Many test comments and names still talk about "sequential" and "parallel" routing and MIN_FILES_FOR_PARALLEL. Examples: the _sequential variants in test_pipeline.c, and the probe files test_edge_types_probe.c, test_convergence_probe.c and test_lang_contract.c.
    • Those tests still pass and still test something useful: a small fixture against a padded one, or one worker against several. Renaming them and rewriting their comments is mechanical and would roughly double this diff, so I left it for a follow-up.

…e had

The parallel path indexes every repository with more than 50 files, but
several results only the sequential path produced:

- A class base that resolves to an Interface is IMPLEMENTS, not INHERITS
  (Java `implements`, C# `: IFace`, TS `implements`). Mirrors 791b768.
- FastAPI `Depends(fn)` in a signature becomes a CALLS edge
  (strategy fastapi_depends) from the endpoint to `fn`.
- A call to the project's own `fetch` is not HTTP: the arg-URL heuristic
  no longer turns `fetch('/api/x')` into HTTP_CALLS once `fetch` resolved
  to a local definition (#856).
- Cross-file LSP ran only when per-file LSP had fewer invocation rows
  than calls. That count included confidence-0 lsp_unresolved rows, so a
  file whose only call was unresolved skipped it (a cross-file TS method
  call stayed unresolved), and it ignored callable-reference candidates,
  so `accept(handler)` with an imported `handler` never got its
  CALL_REFERENCE. Count only rows the CALLS join uses, and also run when
  reference candidates outnumber reference rows.

Found by forcing the parallel path for every index: these were the
suite's only failures.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
Repositories with 50 files or fewer, CBM_WORKERS=1, and incremental runs
touching 50 files or fewer went through a second, sequential
implementation of extraction and resolution (pass_definitions.c,
pass_calls.c, pass_usages.c, pass_semantic.c, cbm_pipeline_pass_lsp_cross).
Every resolver rule had to be written twice and the copies had drifted.

Every index now runs run_parallel_pipeline, whatever its file or worker
count. Incremental runs extract and resolve changed files in parallel,
and re-resolve the callers of changed files through the new
cbm_parallel_resolve_calls, which uses the same resolve_file_calls.

Deleted: pass_definitions.c, pass_usages.c, cbm_pipeline_pass_lsp_cross,
the sequential half of pass_semantic.c (the Go implicit-interface check
stays), the pkgmap builders only the sequential path used, and the
sequential result cache. pass_calls.c keeps only the two helpers the
parallel resolver calls, as service_bindings.c. pass_k8s.c extracts the
files it needs itself instead of reading the deleted cache.

The parity tests in test_parallel.c now compare one worker with two.
The pass_calls args-formatter test goes with its formatter; the parallel
one is covered by parallel_args_json_no_overflow. CBM_WORKERS=0 already
meant one worker; its comment and docs no longer mention a sequential
path.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
An unresolved call whose name ends like a route method, with no route
path (`request.headers.get('content-type')`, `unified().use(plugin)`),
goes through the callee_suffix fallback, which hands emit_service_edge
the caller as the target. With no route found, its plain CALLS
fall-through minted a caller->caller self-loop. The sequential resolver
never emitted it; with every index on the parallel path, small
repositories started getting these edges, as large ones already did.

Suppress the plain CALLS fall-through at that call site. Route
registration, HTTP classification and the arg-URL heuristic are
unchanged.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
@ata-sesli ata-sesli added this to the v2.0.1 milestone Nov 3, 2026
@ata-sesli
ata-sesli added this pull request to stack #120 November 3, 2026 20:14
ata-sesli added a commit that referenced this pull request Nov 3, 2026
#118 removed the sequential pipeline and MIN_FILES_FOR_PARALLEL, so
every index runs the parallel path whatever its file or worker count.
Test names and comments still described a 50-file routing threshold
and a sequential driver. The tests still compare something useful (one
worker against four, or a small fixture against a padded one), so only
names, comments and diagnostic strings change:

- *_sequential tests are now *_one_worker or *_small, and
  *_sequential_parallel tests *_one_and_four_workers;
  pipeline_seq_ts_cross_uses_shared_registry drops the "seq", and
  repro_seq_parallel_equivalence is repro_one_and_many_workers_equivalence.
- "Pad past 50 files so the parallel path runs" becomes "padded".
- probe_hcl_infra_maps_edge was commented as an expected failure
  because one file never reached the parallel path; it already passes.
- src comments in pass_lsp_cross, pipeline.h, pipeline_incremental.c,
  registry.c and graph_buffer.c no longer name a sequential driver.

Comments that describe the sequential pipeline as history are kept.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
@ata-sesli
ata-sesli merged commit dd8517d into main Nov 3, 2026
10 checks passed
ata-sesli added a commit that referenced this pull request Nov 3, 2026
#118 removed the sequential pipeline and MIN_FILES_FOR_PARALLEL, so
every index runs the parallel path whatever its file or worker count.
Test names and comments still described a 50-file routing threshold
and a sequential driver. The tests still compare something useful (one
worker against four, or a small fixture against a padded one), so only
names, comments and diagnostic strings change:

- *_sequential tests are now *_one_worker or *_small, and
  *_sequential_parallel tests *_one_and_four_workers;
  pipeline_seq_ts_cross_uses_shared_registry drops the "seq", and
  repro_seq_parallel_equivalence is repro_one_and_many_workers_equivalence.
- "Pad past 50 files so the parallel path runs" becomes "padded".
- probe_hcl_infra_maps_edge was commented as an expected failure
  because one file never reached the parallel path; it already passes.
- src comments in pass_lsp_cross, pipeline.h, pipeline_incremental.c,
  registry.c and graph_buffer.c no longer name a sequential driver.

Comments that describe the sequential pipeline as history are kept.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant