Skip to content

refactor: share duplicated C and Zig helpers through foundation - #119

Merged
ata-sesli merged 4 commits into
simplify/single-pipeline-pathfrom
simplify/shared-helpers
Nov 3, 2026
Merged

ata-sesli merged 4 commits into
simplify/single-pipeline-pathfrom
simplify/shared-helpers

Conversation

@ata-sesli

@ata-sesli ata-sesli commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Implements #113. Small helpers that were copied between source files now have one definition each: the C ones in src/foundation/, the Zig ones in a new src/foundation/interop.zig. Where a copy behaved differently on purpose, that difference now lives in one named function built on the shared helper.

Stacked on #118. This PR's base is simplify/single-pipeline-path. Review and merge #116, #117 and #118 first; GitHub then retargets this PR to main.

Two commits share the C helpers, then the Zig helpers; two more clean up after #118 (see below). No behaviour change is intended. The output of indexing and queries should be identical.

Problem

Each copy was static, so a fix to one did not reach the others, and some had already drifted:

  • The int-to-string log helpers had ring sizes of 1, 2, 4 or 16 depending on the file.
  • heap_strdup in graph_buffer.c returned "{}" for NULL, while the copies in mcp.c and store.c returned NULL.

#118 deleted the sequential pipeline, and with it the read_file, build_import_map / free_import_map copies and most of the itoa_log copies the issue listed. This PR handles the rest.

C changes

Helper Copies removed Now
int-to-string for log arguments (itoa_log, itoa_buf, itoa_cx, itoa_k8s, pkgmap_itoa) 11 cbm_log_itoa in log.h. Thread-local ring of 16 buffers, the largest any copy had; the most any call site uses at once is 4.
heap_strdup 3 cbm_heap_strdup in compat.h (NULL → NULL). graph_buffer.c keeps its "{}" default as dup_or_empty_json, a one-line wrapper.
intptr_to_ptr 3 cbm_intptr_to_ptr, static inline in compat.h.
elapsed_ms 2 cbm_elapsed_ms in compat.h.
stat_mtime_ns 2 cbm_stat_mtime_ns in compat_fs.h.
path_basename, k8s_basename 3 The existing cbm_path_base (pass_configlink.c, pass_k8s.c, internal/cbm/helpers.c).
path_basename in registry.c — Kept as native_path_basename: cbm_path_base, plus splitting on \ on Windows.
file_ext 2 cbm_path_last_dot in str_util.h.
file_ext in store.c — Kept as file_ext_lower, because it lowercases into a buffer; it now finds the dot with cbm_path_last_dot.
ends_with 2 The existing cbm_str_ends_with. pass_githistory.c's copy took a precomputed length; every caller passed strlen(path).

Why a new cbm_path_last_dot instead of the existing cbm_path_ext: the file_ext copies in pass_similarity.c and pass_semantic_edges.c compare the text from the last . anywhere in the path. cbm_path_ext ignores dots in directory names and strips the dot, so for a path like conf.d/Makefile the result would change. That would change which functions count as having the same extension, and with it SIMILAR_TO and SEMANTICALLY_RELATED edges.

Also removed: the ring-size enums and the duplicate time constants (CBM_NS_PER_SEC, CBM_MS_PER_SEC, CBM_US_PER_SEC_F and others) that only these helpers used. pass_parallel.c used its old ring size PP_RING to size a padding field. That size now has its own constant, PP_RESOLVE_COUNTERS, with the same value of 5.

Zig changes

New src/foundation/interop.zig with the C-interop basics the ported modules each defined for themselves:

  • logging: log, logInfo, logWarn (from cypher.zig, watcher.zig, userconfig.zig);
  • formatting: bprint (from parser.zig and discover.zig; this is parser.zig's version, which also handles an empty buffer);
  • allocation and C strings, formerly pub in parser.zig: mallocN, callocN, reallocN, safeReallocN, strFree, heapStrdup, sp, ciEq. heapStrdup is now the C cbm_heap_strdup, so the Zig copy is gone as well.

cypher.zig imports these from interop.zig instead of parser.zig. The cypher-specific helpers (ix, narrow, isScalarValueFunc) stay in parser.zig. interop.zig is added to CBM_ZIG_SRCS in Makefile.cbm; build.zig finds it through @import.

Follow-ups from #118

Two more commits clean up what #118 left behind.

test: drop sequential-path wording left after the pipeline unification. #118 removed the sequential pipeline and MIN_FILES_FOR_PARALLEL, but test names and comments still described a 50-file routing threshold and a sequential driver. The tests themselves still compare something useful, so they keep their assertions; only names, comments and diagnostic strings change.

  • Renamed tests (old → new):
    • pipeline_issue54_*_sequential (8 tests) → pipeline_issue54_*_one_worker
    • pipeline_folded_topics_preserve_identity_sequential_parallel → ..._one_and_four_workers
    • pipeline_external_import_binding_sequential_parallel → ..._one_and_four_workers
    • pipeline_c_variants_on_sequential_and_parallel_paths_c1 → pipeline_c_variants_with_one_and_four_workers_c1
    • pipeline_rust_grouped_import_properties_valid_json_sequential → ..._one_worker
    • pipeline_typed_http_call_has_no_arg_url_twin_sequential and pipeline_capitalized_axios_import_is_http_client_sequential → ..._small
    • pipeline_seq_ts_cross_uses_shared_registry → pipeline_ts_cross_uses_shared_registry
    • handles_fiber_group_prefix_sequential_issue686 → handles_fiber_group_prefix_small_issue686
    • repro_seq_parallel_equivalence → repro_one_and_many_workers_equivalence
  • Comments that said "pad past 50 files so the parallel path runs" now just say the fixture is padded. Comments that described the sequential resolver as current now describe it as deleted, or drop it.
  • probe_hcl_infra_maps_edge was commented as an expected failure, because a one-file fixture used to skip the parallel path, the only one that emitted INFRA_MAPS. Every index now runs that path, and the assertion already passes. The comment now says so.
  • A few src/ comments said the same: pass_lsp_cross.{c,h}, pipeline.h, pipeline_incremental.c, registry.c and graph_buffer.c.

Comments that describe the sequential pipeline as history ("before this fix, the sequential resolver…") are unchanged. So are uses of "sequential" that mean single-threaded, such as the gbuf merge.

style(watcher): satisfy zig fmt. One if in src/watcher/watcher.zig was not zig fmt-clean on main. The command format string is now a named const, so zig fmt keeps the brace on the if line.

Not changed

  • itoa_imp in pass_importance.c formats a uint64_t, not an int, so it is not a copy of cbm_log_itoa.
  • dupZ in watcher.zig returns a sentinel-terminated Zig pointer rather than a C pointer, so it was left as is.

Verification

  • Full test suite with CBM_TEST_TIERS=all (macOS arm64): 6,449 passed, 0 failed, the same count as refactor(pipeline): run the parallel path for every index #118. The probes' UNEXPECTED PASS notes are identical to refactor(pipeline): run the parallel path for every index #118's.
  • Full suite again after the refactor(pipeline): run the parallel path for every index #118 follow-ups: 6,450 passed, 1 failed. The failure was index_recovery_parallel_quarantines_crasher (an innocent file quarantined). That test depends on which files are in flight when a worker crashes, and this run shared the machine with another indexing process. The mcp suite rerun alone passed (176, including that test). The two earlier full runs passed it too, and these commits change only a comment in test_mcp.c. The total, 6,451, is the 6,449 above plus the two new unit tests. The probes' UNEXPECTED PASS notes are unchanged.
  • New unit tests: log_itoa_keeps_sixteen_values (16 results from consecutive calls stay intact) and path_last_dot (including the directory-dot and NULL cases). Both pass; the log and str_util suites pass (65 tests). They were added after the full run above, so they are not in its count.
  • Build sources: scripts/check-build-sources.sh passes.
  • Format: clang-format-diff (clang-format 22.1.8) is clean on the changed lines. zig fmt --check (Zig 0.17.0) is clean on every changed Zig file, including watcher.zig.
  • Other checks: git diff --check and tests/test_github_workflows.sh pass.

Diff (first two commits): 36 files, +563 / −713.

  • Not checked: the Windows-only branches (native_path_basename and the Windows case of cbm_stat_mtime_ns) were compiled only by reading; there is no Windows build here. Each is a direct move of the old code.

Small helpers were copied as static functions across the pipeline,
store, MCP and graph-buffer sources, and some copies had drifted (ring
sizes of 1 to 16 for the log int-to-string helpers, NULL vs "{}" for
heap_strdup). Each now has one definition in src/foundation:

- cbm_log_itoa (log.h): replaces itoa_log, itoa_buf, itoa_cx, itoa_k8s
  and pkgmap_itoa; thread-local ring of 16 buffers.
- cbm_heap_strdup, cbm_intptr_to_ptr, cbm_elapsed_ms (compat.h).
- cbm_stat_mtime_ns (compat_fs.h).
- cbm_path_last_dot (str_util.h): the file_ext copies, which take the
  text from the last dot anywhere in the path.
- path_basename, k8s_basename and ends_with use the existing
  cbm_path_base and cbm_str_ends_with.

Intentional differences stay in one named function each:
dup_or_empty_json (graph_buffer.c), native_path_basename (registry.c)
and file_ext_lower (store.c).

Implements #113.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
cypher.zig, watcher.zig and userconfig.zig each defined their own
cbm_log wrappers, discover.zig and parser.zig each had bprint, and
cypher.zig imported the allocation and C-string helpers from
parser.zig. These now live in src/foundation/interop.zig. heapStrdup
is the C cbm_heap_strdup, so the Zig copy is gone too.

Implements #113.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
@ata-sesli ata-sesli added this to the v2.0.1 milestone Nov 3, 2026
@ata-sesli
ata-sesli added this pull request to stack #120 November 3, 2026 20:14
The submodule status command's format string, written inline as the
popenGit argument, made zig fmt 0.17 move the if's brace to its own
line. Name the string as a const so the call fits on one line.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
#118 removed the sequential pipeline and MIN_FILES_FOR_PARALLEL, so
every index runs the parallel path whatever its file or worker count.
Test names and comments still described a 50-file routing threshold
and a sequential driver. The tests still compare something useful (one
worker against four, or a small fixture against a padded one), so only
names, comments and diagnostic strings change:

- *_sequential tests are now *_one_worker or *_small, and
  *_sequential_parallel tests *_one_and_four_workers;
  pipeline_seq_ts_cross_uses_shared_registry drops the "seq", and
  repro_seq_parallel_equivalence is repro_one_and_many_workers_equivalence.
- "Pad past 50 files so the parallel path runs" becomes "padded".
- probe_hcl_infra_maps_edge was commented as an expected failure
  because one file never reached the parallel path; it already passes.
- src comments in pass_lsp_cross, pipeline.h, pipeline_incremental.c,
  registry.c and graph_buffer.c no longer name a sequential driver.

Comments that describe the sequential pipeline as history are kept.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
@ata-sesli
ata-sesli merged commit 58e8328 into main Nov 3, 2026
2 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant