Repository navigation
refactor: share duplicated C and Zig helpers through foundation - #119
Merged
Merged
Conversation
Small helpers were copied as static functions across the pipeline,
store, MCP and graph-buffer sources, and some copies had drifted (ring
sizes of 1 to 16 for the log int-to-string helpers, NULL vs "{}" for
heap_strdup). Each now has one definition in src/foundation:
- cbm_log_itoa (log.h): replaces itoa_log, itoa_buf, itoa_cx, itoa_k8s
and pkgmap_itoa; thread-local ring of 16 buffers.
- cbm_heap_strdup, cbm_intptr_to_ptr, cbm_elapsed_ms (compat.h).
- cbm_stat_mtime_ns (compat_fs.h).
- cbm_path_last_dot (str_util.h): the file_ext copies, which take the
text from the last dot anywhere in the path.
- path_basename, k8s_basename and ends_with use the existing
cbm_path_base and cbm_str_ends_with.
Intentional differences stay in one named function each:
dup_or_empty_json (graph_buffer.c), native_path_basename (registry.c)
and file_ext_lower (store.c).
Implements #113.
Signed-off-by: ata-sesli <atasesli@protonmail.com>
cypher.zig, watcher.zig and userconfig.zig each defined their own cbm_log wrappers, discover.zig and parser.zig each had bprint, and cypher.zig imported the allocation and C-string helpers from parser.zig. These now live in src/foundation/interop.zig. heapStrdup is the C cbm_heap_strdup, so the Zig copy is gone too. Implements #113. Signed-off-by: ata-sesli <atasesli@protonmail.com>
The submodule status command's format string, written inline as the popenGit argument, made zig fmt 0.17 move the if's brace to its own line. Name the string as a const so the call fits on one line. Signed-off-by: ata-sesli <atasesli@protonmail.com>
#118 removed the sequential pipeline and MIN_FILES_FOR_PARALLEL, so every index runs the parallel path whatever its file or worker count. Test names and comments still described a 50-file routing threshold and a sequential driver. The tests still compare something useful (one worker against four, or a small fixture against a padded one), so only names, comments and diagnostic strings change: - *_sequential tests are now *_one_worker or *_small, and *_sequential_parallel tests *_one_and_four_workers; pipeline_seq_ts_cross_uses_shared_registry drops the "seq", and repro_seq_parallel_equivalence is repro_one_and_many_workers_equivalence. - "Pad past 50 files so the parallel path runs" becomes "padded". - probe_hcl_infra_maps_edge was commented as an expected failure because one file never reached the parallel path; it already passes. - src comments in pass_lsp_cross, pipeline.h, pipeline_incremental.c, registry.c and graph_buffer.c no longer name a sequential driver. Comments that describe the sequential pipeline as history are kept. Signed-off-by: ata-sesli <atasesli@protonmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #113. Small helpers that were copied between source files now have one definition each: the C ones in
src/foundation/, the Zig ones in a newsrc/foundation/interop.zig. Where a copy behaved differently on purpose, that difference now lives in one named function built on the shared helper.Stacked on #118. This PR's base is
simplify/single-pipeline-path. Review and merge #116, #117 and #118 first; GitHub then retargets this PR tomain.Two commits share the C helpers, then the Zig helpers; two more clean up after #118 (see below). No behaviour change is intended. The output of indexing and queries should be identical.
Problem
Each copy was
static, so a fix to one did not reach the others, and some had already drifted:heap_strdupingraph_buffer.creturned"{}"forNULL, while the copies inmcp.candstore.creturnedNULL.#118 deleted the sequential pipeline, and with it the
read_file,build_import_map/free_import_mapcopies and most of theitoa_logcopies the issue listed. This PR handles the rest.C changes
itoa_log,itoa_buf,itoa_cx,itoa_k8s,pkgmap_itoa)cbm_log_itoainlog.h. Thread-local ring of 16 buffers, the largest any copy had; the most any call site uses at once is 4.heap_strdupcbm_heap_strdupincompat.h(NULL→NULL).graph_buffer.ckeeps its"{}"default asdup_or_empty_json, a one-line wrapper.intptr_to_ptrcbm_intptr_to_ptr,static inlineincompat.h.elapsed_mscbm_elapsed_msincompat.h.stat_mtime_nscbm_stat_mtime_nsincompat_fs.h.path_basename,k8s_basenamecbm_path_base(pass_configlink.c,pass_k8s.c,internal/cbm/helpers.c).path_basenameinregistry.cnative_path_basename:cbm_path_base, plus splitting on\on Windows.file_extcbm_path_last_dotinstr_util.h.file_extinstore.cfile_ext_lower, because it lowercases into a buffer; it now finds the dot withcbm_path_last_dot.ends_withcbm_str_ends_with.pass_githistory.c's copy took a precomputed length; every caller passedstrlen(path).Why a new
cbm_path_last_dotinstead of the existingcbm_path_ext: thefile_extcopies inpass_similarity.candpass_semantic_edges.ccompare the text from the last.anywhere in the path.cbm_path_extignores dots in directory names and strips the dot, so for a path likeconf.d/Makefilethe result would change. That would change which functions count as having the same extension, and with it SIMILAR_TO and SEMANTICALLY_RELATED edges.Also removed: the ring-size enums and the duplicate time constants (
CBM_NS_PER_SEC,CBM_MS_PER_SEC,CBM_US_PER_SEC_Fand others) that only these helpers used.pass_parallel.cused its old ring sizePP_RINGto size a padding field. That size now has its own constant,PP_RESOLVE_COUNTERS, with the same value of 5.Zig changes
New
src/foundation/interop.zigwith the C-interop basics the ported modules each defined for themselves:log,logInfo,logWarn(fromcypher.zig,watcher.zig,userconfig.zig);bprint(fromparser.ziganddiscover.zig; this isparser.zig's version, which also handles an empty buffer);pubinparser.zig:mallocN,callocN,reallocN,safeReallocN,strFree,heapStrdup,sp,ciEq.heapStrdupis now the Ccbm_heap_strdup, so the Zig copy is gone as well.cypher.zigimports these frominterop.ziginstead ofparser.zig. The cypher-specific helpers (ix,narrow,isScalarValueFunc) stay inparser.zig.interop.zigis added toCBM_ZIG_SRCSinMakefile.cbm;build.zigfinds it through@import.Follow-ups from #118
Two more commits clean up what #118 left behind.
test: drop sequential-path wording left after the pipeline unification. #118 removed the sequential pipeline andMIN_FILES_FOR_PARALLEL, but test names and comments still described a 50-file routing threshold and a sequential driver. The tests themselves still compare something useful, so they keep their assertions; only names, comments and diagnostic strings change.pipeline_issue54_*_sequential(8 tests) →pipeline_issue54_*_one_workerpipeline_folded_topics_preserve_identity_sequential_parallel→..._one_and_four_workerspipeline_external_import_binding_sequential_parallel→..._one_and_four_workerspipeline_c_variants_on_sequential_and_parallel_paths_c1→pipeline_c_variants_with_one_and_four_workers_c1pipeline_rust_grouped_import_properties_valid_json_sequential→..._one_workerpipeline_typed_http_call_has_no_arg_url_twin_sequentialandpipeline_capitalized_axios_import_is_http_client_sequential→..._smallpipeline_seq_ts_cross_uses_shared_registry→pipeline_ts_cross_uses_shared_registryhandles_fiber_group_prefix_sequential_issue686→handles_fiber_group_prefix_small_issue686repro_seq_parallel_equivalence→repro_one_and_many_workers_equivalenceprobe_hcl_infra_maps_edgewas commented as an expected failure, because a one-file fixture used to skip the parallel path, the only one that emitted INFRA_MAPS. Every index now runs that path, and the assertion already passes. The comment now says so.src/comments said the same:pass_lsp_cross.{c,h},pipeline.h,pipeline_incremental.c,registry.candgraph_buffer.c.Comments that describe the sequential pipeline as history ("before this fix, the sequential resolver…") are unchanged. So are uses of "sequential" that mean single-threaded, such as the gbuf merge.
style(watcher): satisfy zig fmt. Oneifinsrc/watcher/watcher.zigwas notzig fmt-clean onmain. The command format string is now a namedconst, sozig fmtkeeps the brace on theifline.Not changed
itoa_impinpass_importance.cformats auint64_t, not anint, so it is not a copy ofcbm_log_itoa.dupZinwatcher.zigreturns a sentinel-terminated Zig pointer rather than a C pointer, so it was left as is.Verification
CBM_TEST_TIERS=all(macOS arm64): 6,449 passed, 0 failed, the same count as refactor(pipeline): run the parallel path for every index #118. The probes' UNEXPECTED PASS notes are identical to refactor(pipeline): run the parallel path for every index #118's.index_recovery_parallel_quarantines_crasher(an innocent file quarantined). That test depends on which files are in flight when a worker crashes, and this run shared the machine with another indexing process. Themcpsuite rerun alone passed (176, including that test). The two earlier full runs passed it too, and these commits change only a comment intest_mcp.c. The total, 6,451, is the 6,449 above plus the two new unit tests. The probes' UNEXPECTED PASS notes are unchanged.log_itoa_keeps_sixteen_values(16 results from consecutive calls stay intact) andpath_last_dot(including the directory-dot andNULLcases). Both pass; thelogandstr_utilsuites pass (65 tests). They were added after the full run above, so they are not in its count.scripts/check-build-sources.shpasses.clang-format-diff(clang-format 22.1.8) is clean on the changed lines.zig fmt --check(Zig 0.17.0) is clean on every changed Zig file, includingwatcher.zig.git diff --checkandtests/test_github_workflows.shpass.Diff (first two commits): 36 files, +563 / −713.
native_path_basenameand the Windows case ofcbm_stat_mtime_ns) were compiled only by reading; there is no Windows build here. Each is a direct move of the old code.