Repository navigation
refactor(pipeline): run the parallel path for every index - #118
Merged
Merged
Conversation
…e had The parallel path indexes every repository with more than 50 files, but several results only the sequential path produced: - A class base that resolves to an Interface is IMPLEMENTS, not INHERITS (Java `implements`, C# `: IFace`, TS `implements`). Mirrors 791b768. - FastAPI `Depends(fn)` in a signature becomes a CALLS edge (strategy fastapi_depends) from the endpoint to `fn`. - A call to the project's own `fetch` is not HTTP: the arg-URL heuristic no longer turns `fetch('/api/x')` into HTTP_CALLS once `fetch` resolved to a local definition (#856). - Cross-file LSP ran only when per-file LSP had fewer invocation rows than calls. That count included confidence-0 lsp_unresolved rows, so a file whose only call was unresolved skipped it (a cross-file TS method call stayed unresolved), and it ignored callable-reference candidates, so `accept(handler)` with an imported `handler` never got its CALL_REFERENCE. Count only rows the CALLS join uses, and also run when reference candidates outnumber reference rows. Found by forcing the parallel path for every index: these were the suite's only failures. Signed-off-by: ata-sesli <atasesli@protonmail.com>
Repositories with 50 files or fewer, CBM_WORKERS=1, and incremental runs touching 50 files or fewer went through a second, sequential implementation of extraction and resolution (pass_definitions.c, pass_calls.c, pass_usages.c, pass_semantic.c, cbm_pipeline_pass_lsp_cross). Every resolver rule had to be written twice and the copies had drifted. Every index now runs run_parallel_pipeline, whatever its file or worker count. Incremental runs extract and resolve changed files in parallel, and re-resolve the callers of changed files through the new cbm_parallel_resolve_calls, which uses the same resolve_file_calls. Deleted: pass_definitions.c, pass_usages.c, cbm_pipeline_pass_lsp_cross, the sequential half of pass_semantic.c (the Go implicit-interface check stays), the pkgmap builders only the sequential path used, and the sequential result cache. pass_calls.c keeps only the two helpers the parallel resolver calls, as service_bindings.c. pass_k8s.c extracts the files it needs itself instead of reading the deleted cache. The parity tests in test_parallel.c now compare one worker with two. The pass_calls args-formatter test goes with its formatter; the parallel one is covered by parallel_args_json_no_overflow. CBM_WORKERS=0 already meant one worker; its comment and docs no longer mention a sequential path. Signed-off-by: ata-sesli <atasesli@protonmail.com>
An unresolved call whose name ends like a route method, with no route
path (`request.headers.get('content-type')`, `unified().use(plugin)`),
goes through the callee_suffix fallback, which hands emit_service_edge
the caller as the target. With no route found, its plain CALLS
fall-through minted a caller->caller self-loop. The sequential resolver
never emitted it; with every index on the parallel path, small
repositories started getting these edges, as large ones already did.
Suppress the plain CALLS fall-through at that call site. Route
registration, HTTP classification and the arg-URL heuristic are
unchanged.
Signed-off-by: ata-sesli <atasesli@protonmail.com>
ata-sesli
added a commit
that referenced
this pull request
Nov 3, 2026
#118 removed the sequential pipeline and MIN_FILES_FOR_PARALLEL, so every index runs the parallel path whatever its file or worker count. Test names and comments still described a 50-file routing threshold and a sequential driver. The tests still compare something useful (one worker against four, or a small fixture against a padded one), so only names, comments and diagnostic strings change: - *_sequential tests are now *_one_worker or *_small, and *_sequential_parallel tests *_one_and_four_workers; pipeline_seq_ts_cross_uses_shared_registry drops the "seq", and repro_seq_parallel_equivalence is repro_one_and_many_workers_equivalence. - "Pad past 50 files so the parallel path runs" becomes "padded". - probe_hcl_infra_maps_edge was commented as an expected failure because one file never reached the parallel path; it already passes. - src comments in pass_lsp_cross, pipeline.h, pipeline_incremental.c, registry.c and graph_buffer.c no longer name a sequential driver. Comments that describe the sequential pipeline as history are kept. Signed-off-by: ata-sesli <atasesli@protonmail.com>
ata-sesli
added a commit
that referenced
this pull request
Nov 3, 2026
#118 removed the sequential pipeline and MIN_FILES_FOR_PARALLEL, so every index runs the parallel path whatever its file or worker count. Test names and comments still described a 50-file routing threshold and a sequential driver. The tests still compare something useful (one worker against four, or a small fixture against a padded one), so only names, comments and diagnostic strings change: - *_sequential tests are now *_one_worker or *_small, and *_sequential_parallel tests *_one_and_four_workers; pipeline_seq_ts_cross_uses_shared_registry drops the "seq", and repro_seq_parallel_equivalence is repro_one_and_many_workers_equivalence. - "Pad past 50 files so the parallel path runs" becomes "padded". - probe_hcl_infra_maps_edge was commented as an expected failure because one file never reached the parallel path; it already passes. - src comments in pass_lsp_cross, pipeline.h, pipeline_incremental.c, registry.c and graph_buffer.c no longer name a sequential driver. Comments that describe the sequential pipeline as history are kept. Signed-off-by: ata-sesli <atasesli@protonmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #110. Every index, full or incremental, now goes through the parallel extract/resolve path, whatever its file count or worker count. The sequential passes are deleted, and the parallel resolver gained the results that only the sequential one used to produce.
Stacked on #117. This PR's base is
simplify/zova-ops-table. Review and merge #116 and #117 first; GitHub then retargets this PR tomain.Problem
Indexing had two implementations:
pass_definitions.c,pass_calls.c,pass_usages.c,pass_semantic.candcbm_pipeline_pass_lsp_cross. It ran for repositories with 50 files or fewer, forCBM_WORKERS=1, and for incremental runs touching 50 files or fewer.pass_parallel.c. It ran for everything else.Each resolver rule had to be written twice, and "keep in sync" comments asked for that by hand. The two copies had drifted. The same project could get different edges depending on how many files it had or how many workers ran it.
Change
Three commits.
1.
fix(pipeline): give the parallel resolver what only the sequential one hadI forced the parallel path for every index and ran the full suite. These were the only failures, and this commit fixes them in
pass_parallel.c:Depends(fn). It becomes a CALLS edge (strategy: fastapi_depends).fetch. A call to the project's ownfetchis no longer turned into HTTP_CALLS by the arg-URL heuristic (#856).lsp_unresolvedrows, so files whose only call was unresolved skipped cross-file LSP. It also ignored callable-reference candidates. It now counts only the rows the CALLS join uses, and also runs when reference candidates outnumber reference rows.2.
refactor(pipeline): run the parallel path for every indexPipeline routing:
pipeline.calways runsrun_parallel_pipeline.run_sequential_pipelineandMIN_FILES_FOR_PARALLELare gone.pipeline_incremental.calways extracts and resolves changed files in parallel.cbm_parallel_resolve_callsinpass_parallel.c, which uses the sameresolve_file_callsas a full index.Deleted:
pass_definitions.candpass_usages.c.cbm_pipeline_pass_lsp_cross.pass_semantic.c. The Go implicit-interface check remains in that file.cbm_pkgmap_build_from_repo/cbm_pkgmap_build_from_files.Moved and kept:
pass_calls.cis nowservice_bindings.c. It keeps only the two helpers the parallel resolver calls:cbm_pipeline_http_client_call_url(#1916) andcbm_pipeline_external_import_service.pass_k8s.calways extracts kustomize and manifest files itself; it no longer reads them from the deleted cache.CBM_WORKERS=0already meant one worker. The comment anddocs/CONFIGURATION.mdno longer say it selects a sequential path.Tests:
test_parallel.c. They compared the deleted passes against the parallel path. They now compare a 1-worker run against a 2-worker run.pipeline_calls_args_overlong_argument_stops_cleanlytested a formatter that only existed inpass_calls.c;parallel_args_json_no_overflowcovers the parallel one.typed_http_no_url_twin_casewas skipped for the small fixture because the sequential resolver lacked the heuristic. It now runs for both fixture sizes.3.
fix(pipeline): drop the self-loop of an unresolved route-suffix callSmall-repo parity found one more gap. An unresolved call whose name ends like a route method (
request.headers.get('content-type'),unified().use(plugin)) with no route path goes through thecallee_suffixfallback. That fallback passes the caller as the target, so its plain CALLS fall-through made a caller→caller self-loop. The sequential resolver never emitted it. Large repos onmainalready had these self-loops, because they took the parallel path.New test:
pipeline_unresolved_route_suffix_call_adds_no_self_loop. It fails without the fix.Diff: 30 files, about 3,000 fewer lines.
Output compared with
mainMeasured with the CLI (
index_repositoryinto a fresh cache),mainat 616926c vs this branch, macOS arm64. Counts are per node label and per edge type fromget_graph_schema.medium-cli(TypeScript)mini-observer(C, Python)main)The two small repositories took the sequential path on
main. They give identical counts, both with the default worker count and withCBM_WORKERS=1.The large-repository differences are expected:
httpGetandfetchChecksumsin the Go installer).same_moduletolsp_direct, because the cross-file LSP gate in commit 1 now runs for that file.tests/test_c_lsp.cagainst other LSP test files) trading one nearest neighbour for another.The other commit 1 fixes add or retype edges only for code this repository does not have: Java/C#/TS interfaces, FastAPI, and a local
fetch. On other projects, expect more IMPLEMENTS (fewer INHERITS), newfastapi_dependsCALLS, more cross-file LSP CALLS / CALL_REFERENCE, and no HTTP_CALLS for calls to a project's ownfetch.Index time
Wall time of the
cli index_repositoryprocess; 5 runs per small case, 3 for the large one; median shown.mainmedium-climedium-climini-observermini-observerSmall repositories are as fast or faster on the parallel path. That includes one worker, which used to mean the sequential path.
The large repository is about 2.6% slower (0.45 s over 3 runs each). The likely cause is the wider cross-file LSP gate from commit 1. Peak RSS was 1.11–1.20 GB on
mainand 1.21–1.28 GB on this branch.Verification
CBM_TEST_TIERS=all(macOS arm64): 6,449 passed, 0 failed.pass_callsformatter test, plus the new self-loop test.pipeline_unresolved_route_suffix_call_adds_no_self_loopfailed withASSERT(!(self_loop))before commit 3.scripts/check-build-sources.shpasses after the build-list changes.clang-format-diff(clang-format 22.1.8) is clean on the changed lines;git diff --checkandtests/test_github_workflows.shpass.service_bindings.cis the renamedpass_calls.c, and one signature carried over unchanged frommainis wrapped differently from clang-format's output. I left it as onmain.Not done
MIN_FILES_FOR_PARALLEL. Examples: the_sequentialvariants intest_pipeline.c, and the probe filestest_edge_types_probe.c,test_convergence_probe.candtest_lang_contract.c.