Reuse immutable default catalogs across library runs

.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
     hooks:
       - id: check-added-large-files
         # Retain the exact lossless cross-platform performance evidence archives.
-        exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff)-20261102)\.json\.gz$
+        exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs)-20261102)\.json\.gz$
       - id: check-merge-conflict
       - id: check-yaml
       # BINARY_TYPES below: archive containers are binary but can START with text (a phar begins with

CHANGELOG.md

@@ -3,6 +3,8 @@
 
 # 0.11.0
 
+- Reuse immutable default language and MIME catalogs across repeated library runs while preserving custom configurations and later registrations.
+
 - Reuse requested hashes and line counts from content already loaded by workers, without retaining file bytes between stages.
 
 - Limit warmed matcher participation to available work chunks while retaining useful read concurrency and persistent workers.

TODO.md

@@ -416,7 +416,11 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms
       [Cross-platform evidence](docs/benchmarks/derived-content-handoff-20261102.md) retains
       read-count, lifetime/error/limit tests, large field-workload gains and slower controls.
       Transfer requested hashes/line values only; keep raw bytes confined to each evaluator.
-- [ ] O14-O22: Reconcile remaining static catalogs, invariant operands,
+- [x] O14: Reuse immutable generated defaults across repeated language/MIME configurations,
+      refreshing after append-only registration and preserving custom overlays and old views.
+      [Warm construction, RSS and native controls](docs/benchmarks/immutable-default-catalogs-20261102.md)
+      qualify reuse only; first-parse generated tables and per-run MIME binding remain unqualified.
+- [ ] O15-O22: Reconcile remaining invariant operands,
       paired streaming compare, reusable stage pools, byte-bounded overlap, archive indexing/admission,
       optional bulk metadata, similarity representations and output churn with EP/JC/P acceptance.
       Keep macrobenchmark, kernel and diagnostic contracts separate.

bazelmod/clang_tidy_targets.bzl

@@ -17,6 +17,7 @@ CLANG_TIDY_MANUAL_TARGETS = [
     "//xff/engine:match_tail_benchmark",
     "//xff/engine:expression_benchmark",
     "//xff/engine:expression_run_benchmark",
+    "//xff/matching/language:default_cache_benchmark",
     "//xff/matching/regex:regex_benchmark",
     "//xff/matching/similarity:near_duplicate_benchmark",
     "//xff/matching/similarity:similarity_benchmark",

docs/benchmarks/immutable-default-catalogs-20261102.md

@@ -0,0 +1,70 @@
+# Reuse immutable default catalogs (2026-11-02)
+
+## Decision
+
+Retain one generated default vocabulary per append-only database-registration epoch and reuse the default filtering catalog. Keep default initialization lazy. Repeated `Configure` calls without user files/edits can restore the same immutable default rather than reparsing generated JSON, rebuilding maps and retaining another complete snapshot. A later registration invalidates that default by registry size; registration only appends through the existing API. Old views and catalog handles remain valid.
+
+Custom files and type edits still build and validate a fresh configuration before publication. User files are not cached by pathname. Restoring defaults after a custom configuration selects the immutable default again. No public API, flag, per-file branch, new lock or ownership wrapper is added. Existing per-entry global lookup locks remain; this is not a new lock-free lookup representation.
+
+The actual opportunity is repeated library use and retained default memory. Empty configurations were already lazy for ordinary CLI startup. This change does not eliminate the first JSON parse or claim that each fresh CLI starts faster. Generated flat/offset tables, per-run MIME observers and broader catalog binding remain separate unqualified follow-ups. Pointer-rich static tables could add binary size/macOS fixups, as the existing startup audit explains.
+
+## Measurement contracts
+
+Measured parent `ed9763f4f8f893c3fd9d7d727c175b0433fb9ac9` (#1053). Final synchronization preserves that source tree. Clean `clang_release`/ThinLTO baseline and `immutable_defaults` binaries are copied before measurement. All runtime patches, support sources, binary hashes, flags, raw samples and scripts are retained. Copied benchmark binaries load only stable system libraries. No build, lint or sanitizer overlaps timing on the same host.
+
+Fresh-command controls cover 40 native cases per platform in three sessions, a discarded warmup per participant/case/session, nine rotating rounds and fastest-seven means. Independent output oracles validate every invocation. Ten/1,000/10,000-file MIME filters/fields, custom type additions and ordinary listing/metadata/comparison use one/three/ten requested workers. Linux uses tmpfs and CPU affinity; macOS uses warm APFS without affinity.
+
+The three separate repeated-library kernels link the real language/MIME database modules. Each process warms the selected mode once before each repetition, then times exactly five resets/lookups. Nine repetitions and fastest-seven means are measured in three sessions. Fixed iteration counts bound the baseline retained snapshots; adaptive calibration would retain ever more complete JSON vocabularies. Sub-microsecond candidate values include loop/timer costs and are not a fresh-startup or per-file throughput estimate.
+
+The memory probe runs each case/variant in three fresh children, with the same nine repetitions/five resets and one setup reset per repetition (54 resets per child). Peak RSS is reported separately from kernel timing and allocation counts. Five native controls receive five-session confirmation with a byte-identical baseline participant; those results do not replace the initial samples.
+
+## Repeated library construction
+
+Values are microseconds per timed reset+lookup; the first construction is outside the timed loop.
+
+| Mode     | Linux baseline us | Linux reused us | macOS baseline us | macOS reused us |
+| :------- | ----------------: | --------------: | ----------------: | --------------: |
+| language |         2,018.578 |           0.080 |         2,826.899 |           0.238 |
+| mime     |         3,684.825 |           0.104 |         4,764.123 |           0.308 |
+| catalog  |         3,179.203 |           0.142 |         4,833.805 |           0.527 |
+
+Repeated default construction drops from milliseconds to sub-microsecond handle/reset work. The language test independently verifies that repeated defaults call the immutable provider once, custom edits still construct their own configuration, and a later registration refreshes the default. These numbers do not describe the first construction in a new process.
+
+## Peak RSS in the repeated-use probe
+
+MiB, median of three children per case and variant; includes executable/runtime and retained data, not just cache-owned bytes.
+
+| Mode     | Linux baseline MiB | Linux reused MiB | macOS baseline MiB | macOS reused MiB |
+| :------- | -----------------: | ---------------: | -----------------: | ---------------: |
+| language |              97.91 |             5.99 |              90.19 |             4.92 |
+| mime     |              95.35 |             6.42 |              82.16 |             5.45 |
+| catalog  |              98.61 |             6.41 |              89.98 |             5.81 |
+
+The baseline intentionally retains published snapshots to keep existing views valid. Reusing the default avoids that repeated growth while retaining customized and older-registration snapshots. This does not introduce a total memory cap or garbage-collect snapshots still covered by the lifetime contract.
+
+## Fresh CLI controls and confirmation
+
+Median paired session changes; negative means faster. Preserve initial costs and session reversals rather than projecting warm-kernel gains onto native commands.
+
+| Fixture / task        | Workers | Initial Linux | Confirm Linux | Baseline repeat Linux | Initial macOS | Confirm macOS | Baseline repeat macOS |
+| :-------------------- | ------: | ------------: | ------------: | --------------------: | ------------: | ------------: | --------------------: |
+| flat10000 / metadata  |      10 |        +9.88% |        +1.18% |                +0.04% |        -0.19% |        +1.81% |                +0.84% |
+| flat10000 / files     |       1 |        +3.02% |        +0.09% |                -0.29% |        -0.42% |        +0.98% |                +0.27% |
+| flat10 / mime-filter  |      10 |        +2.34% |        -0.46% |                -0.72% |        +0.10% |        +1.42% |                +0.24% |
+| flat10 / metadata     |       3 |        -0.98% |        +0.14% |                -0.13% |        +1.65% |        +0.69% |                +0.01% |
+| flat1000 / mime-field |       1 |        -1.29% |        -0.93% |                -0.84% |        -1.22% |        +1.94% |                -0.07% |
+
+The initial Linux 10,000-file/ten-worker metadata control is +9.88% with session values +16.47/+0.30/+9.88%; confirmation is +1.18% with the repeated baseline +0.04%. Metadata-only listing does not request these catalogs, so the cause must not be attributed to default reuse. The confirmed macOS MIME-field control remains about +1.94% and metadata about +1.81%; retain those modest costs. The warm-memory benefit does not imply every CLI is faster.
+
+## Correctness and footprint
+
+Separate test processes verify reused default-view identity, provider-call counts, custom type edits/default restoration, new registrations, preferred MIME overrides, and old view/catalog validity. Existing core and bundled-database suites retain shared extension candidates, aliases, precedence, metadata and error handling. No new test uses host file I/O.
+
+- linux: unstripped CLI output 12,334,504 -> 12,337,640 bytes (+3,136).
+- macos: unstripped CLI output 4,069,744 -> 4,069,696 bytes (-48).
+
+Complete core/extension release suites pass 308/308 per host, with required Linux FUSE mounts. Seven focused ASan suites pass on each host, seven Linux TSan suites pass, and all five changed translation units pass local scoped clang-tidy. Normal enforcing hooks and current pushed-head CI remain required before merge.
+
+## Lossless evidence
+
+[immutable-default-catalogs-20261102.json.gz](immutable-default-catalogs-20261102.json.gz): 714,698 bytes; SHA-256 `74bdd51eed4d7225a56fab4b192b33a771cc732adfebc784cd43c3e97d77faea`. Recover native fixtures through `fixtures[report.fixtures_ref]`. Initial, confirmation, warm-loop and RSS populations retain separate provenance.

docs/performance-analysis.md

@@ -2919,7 +2919,7 @@ virtual layers and generic executor hierarchies need an actual lifetime or exten
 | O22 | Reduce output construction churn      | Numeric/cell/JSON strings, row insertion and aligned-table retention allocate       | Existing output/scope contracts            | Append into reusable row buffers; stream machine formats     |
 
 O01-O04 and the selected O06-O11 mechanisms have qualified cross-platform reports below.
-O05 and O12 record rejected implementations. Other entries remain hypotheses until their individual report
+O05 and O12 record rejected implementations. O13 and the repeated-default subset of O14 have reports below. Other entries remain hypotheses until their individual report
 records a decision; inspected allocation sites alone do not establish elapsed gains.
 
 ### Small changes first
@@ -3341,3 +3341,24 @@ work, rather than proving that copies alone accounted for that time. The report
 macOS output control (+6.63%) and its confirmation (-0.52%), identical-baseline participants, all
 slower samples, source identities and validation. This does not replace O14/O15 invariant/catalog
 work, O17/O18 scheduling or O22 output construction.
+
+### O14 decision: reuse immutable generated defaults between library runs
+
+[The two-platform report](benchmarks/immutable-default-catalogs-20261102.md) qualifies a narrow
+subset of the startup/catalog plan: retain the generated default vocabulary and filtering catalog
+across empty `Configure` resets. Append-only registration invalidates that default; custom files
+and type edits retain their existing fresh construction and validation. Published old views stay
+valid. Existing global lookup locks and per-entry behavior are unchanged.
+
+The real benefit is repeated library construction and retained memory, not first CLI startup.
+Warm reset/lookup kernels fall from milliseconds to sub-microsecond work, including loop/timer
+costs. Separate 54-reset child probes fall from roughly 82-99 MiB peak RSS to 5-7 MiB. Native
+fresh-process controls do not inherit those warm gains. The initial Linux metadata control's
++9.88% falls to +1.18% on confirmation; modest macOS costs up to about +1.94% remain recorded,
+alongside a byte-identical baseline participant. Full 308-test release suites pass on both hosts;
+focused ASan, Linux TSan and changed-unit clang-tidy pass.
+
+This preserves the earlier lazy-default and macOS fixup findings. Generated flat/offset tables,
+first JSON parsing, per-run MIME observers and broader immutable catalog binding remain follow-up
+hypotheses; do not call them implemented or infer gains from the warm reset kernel. No new
+wrapper, lock, per-file clock, thread or exception mechanism is introduced.

release-site.json

@@ -71,7 +71,8 @@
     "docs/benchmarks/bounded-match-results-20261102.md": "docs/benchmarks/bounded-match-results-20261102.html",
     "docs/benchmarks/useful-match-workers-20261102.md": "docs/benchmarks/useful-match-workers-20261102.html",
     "docs/benchmarks/direct-evaluator-workers-20261102.md": "docs/benchmarks/direct-evaluator-workers-20261102.html",
-    "docs/benchmarks/derived-content-handoff-20261102.md": "docs/benchmarks/derived-content-handoff-20261102.html"
+    "docs/benchmarks/derived-content-handoff-20261102.md": "docs/benchmarks/derived-content-handoff-20261102.html",
+    "docs/benchmarks/immutable-default-catalogs-20261102.md": "docs/benchmarks/immutable-default-catalogs-20261102.html"
   },
   "generated_html": {
     "XFF.md": {

xff/matching/language/BUILD.bazel

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-load("//xff:cc.bzl", "cc_library", "cc_test")
+load("//xff:cc.bzl", "cc_binary", "cc_library", "cc_test")
 
 package(default_visibility = ["//visibility:private"])
 
@@ -85,3 +85,33 @@ cc_test(
         "@xff_language_db//:language_db_register_cc",
     ],
 )
+
+cc_test(
+    name = "default_cache_test",
+    size = "small",
+    srcs = ["default_cache_test.cc"],
+    deps = [
+        ":language_cc",
+        "@googletest//:gtest",
+        "@googletest//:gtest_main",
+        "@mboworks_mbo//mbo/testing:matchers_cc",
+        "@mboworks_mbo//mbo/testing:status_cc",
+        "@xff_extras_api//:language_database_cc",
+    ],
+)
+
+cc_binary(
+    name = "default_cache_benchmark",
+    srcs = ["default_cache_benchmark.cc"],
+    tags = [
+        "clang-tidy",
+        "manual",
+    ],
+    deps = [
+        ":language_cc",
+        "//xff/matching/mime:mime_cc",
+        "@com_github_google_benchmark//:benchmark",
+        "@xff_language_db//:language_db_register_cc",
+        "@xff_mime_db//:mime_db_register_cc",
+    ],
+)

xff/matching/language/default_cache_benchmark.cc

@@ -0,0 +1,66 @@
+// SPDX-FileCopyrightText: Copyright (c) M. Boerger, the MBO Works authors
+// SPDX-License-Identifier: Apache-2.0
+
+#include "benchmark/benchmark.h"
+#include "xff/matching/language/catalog.h"
+#include "xff/matching/language/language.h"
+#include "xff/matching/mime/mime.h"
+
+namespace {
+namespace language = ::xff::language;
+namespace mime = ::xff::mime;
+
+enum class Mode { kLanguage, kMime, kCatalog };
+
+bool ResetAndCheck(Mode mode) {
+  if (mode == Mode::kMime) {
+    return mime::Configure({}, mime::ConflictPolicy::kError).ok() && mime::TypeForName("sample.txt") == "text/plain";
+  }
+  if (!language::Configure({}, language::ConflictPolicy::kError).ok()) {
+    return false;
+  }
+  if (mode == Mode::kCatalog) {
+    return !language::ActiveCatalog().CandidatesForName("main.cc").empty();
+  }
+  return language::ActiveSnapshot().LanguageForName("main.cc") == "C++";
+}
+
+void Reconfigure(benchmark::State& state, Mode mode) {
+  // Warm once before timing. This measures repeated library invocations, not fresh CLI startup.
+  if (!ResetAndCheck(mode)) {
+    state.SkipWithError("incorrect default catalog before measurement");
+    return;
+  }
+  for (auto iteration : state) {
+    benchmark::DoNotOptimize(iteration);
+    bool correct = ResetAndCheck(mode);
+    benchmark::DoNotOptimize(correct);
+    if (!correct) {
+      state.SkipWithError("incorrect reused catalog");
+      break;
+    }
+  }
+  state.SetItemsProcessed(state.iterations());
+}
+}  // namespace
+
+// XFF_ABI_POINTER: Google Benchmark's process argument interface.
+int main(int argc, char** argv) {
+  benchmark::Initialize(&argc, argv);
+  // Fixed iterations bound the baseline's deliberately process-retained snapshots. Adaptive
+  // calibration would retain ever more complete JSON vocabularies in the baseline.
+  benchmark::RegisterBenchmark("reset-default/language", Reconfigure, Mode::kLanguage)
+      ->Iterations(5)
+      ->UseRealTime()
+      ->Unit(benchmark::kMicrosecond);
+  benchmark::RegisterBenchmark("reset-default/mime", Reconfigure, Mode::kMime)
+      ->Iterations(5)
+      ->UseRealTime()
+      ->Unit(benchmark::kMicrosecond);
+  benchmark::RegisterBenchmark("reset-default/catalog", Reconfigure, Mode::kCatalog)
+      ->Iterations(5)
+      ->UseRealTime()
+      ->Unit(benchmark::kMicrosecond);
+  benchmark::RunSpecifiedBenchmarks();
+  benchmark::Shutdown();
+}