experiment(search): productionize hybrid fusion and evaluate HDC

[ajroetker]

Decision

This PR changes the product direction:

• Invest in the hybrid query path.
• Do not promote HDC to a stable public index.

The production goal is to combine lexical, dense, sparse, exact structured, and explicit graph evidence through bounded candidate generation, calibrated fusion, exact candidate scoring, optional reranking, and explainable final scores.

Packed-bipolar HDC is evaluated only as an optional candidate-local structural interaction feature. On the current held-out workload it makes the learned reranker worse, so it remains a no-go.

The existing hypervector prototype stays experimental. If HDC is ever revived, it should be a distinct logical representation/index that lowers to existing dense-vector storage and ANN machinery—not a bare hdc option on ordinary embeddings, not a new physical engine, and not an implicit graph-index creator.

Hybrid path implemented

• RRF and RSF over named lexical, dense, sparse, structured, and graph sources
• non-negative named weights with unknown, duplicate, non-finite, all-zero, and incompatible configurations rejected
• deterministic document-ID tie breaking
• globally rebuilt ranks and RSF calibration across shards
• bounded per-source retrieval and bounded union fanout
• exact with membership scored over the full retrieved candidate union
• explicit graph result sets as weighted ranked sources
• cross-range weighted graph fusion with hydrated result documents
• reranking of the complete fused candidate pool; top_n truncates only after scoring
• explain: true output containing raw score, rank, calibrated score, weight, contribution, fused score, final score, and reranked state
• incomplete candidate windows reported with total.relation: "gte"
• unsupported serverless controls rejected instead of silently changing semantics

Graph-index creation remains explicit and independent.

UX

{
  "full_text_search": {
    "match": {"field": "description", "text": "quiet waterfront city"}
  },
  "semantic_search": "quiet waterfront city with mountain access",
  "indexes": ["location_embedding"],
  "with": {
    "pacific_region": {
      "term": {"path": "/region", "value": "pacific_northwest"}
    }
  },
  "graph_searches": {
    "near_transit_hub": {
      "type": "neighbors",
      "index_name": "travel_graph",
      "start_nodes": {"keys": ["hub:west-coast"]},
      "params": {"edge_types": ["SERVES"], "direction": "out", "max_depth": 1}
    }
  },
  "merge_config": {
    "strategy": "rsf",
    "window_size": 100,
    "weights": {
      "full_text": 0.35,
      "location_embedding": 1.0,
      "pacific_region": 0.2,
      "near_transit_hub": 0.15
    }
  },
  "reranker": {
    "provider": "antfly",
    "model": "cross-encoder/ms-marco-MiniLM-L-6-v2",
    "field": "description",
    "top_n": 10
  },
  "limit": 100,
  "explain": true
}

window_size is the per-ranked-source retrieval/calibration budget. limit is the final page size without a reranker; with a reranker it is the fused candidate pool, and reranker.top_n is the final count. Exact filters, tenant constraints, deletion visibility, and graph facts remain authoritative.

HDC evidence

The pinned WANDS protocol uses:

• 42,994 products
• 480 human-judged queries in the full workload
• a 116-query real class-plus-attribute compositional slice
• deterministic validation/holdout splits
• equal 50/100/200 candidate budgets
• paired 10,000-sample bootstrap intervals
• dense RaBitQ and packed-bipolar HDC representations

The new candidate-local follow-up fixes embedding RaBitQ as candidate generation and compares:

1. transparent semantic + exact structured scoring;
2. a validation-trained pairwise linear reranker; and
3. the same learner with an 8,000-bit packed-bipolar HDC structural feature.

Held-out NDCG@10:

Candidates	Fixed hybrid	Learned	Learned + HDC	HDC delta vs learned (95% CI)
50	0.7472	0.7476	0.7385	-0.009063 [-0.017478, -0.000973]
100	0.7574	0.7599	0.7521	-0.007775 [-0.015767, -0.000181]
200	0.7521	0.7613	0.7541	-0.007121 [-0.016230, 0.001916]

At 200 candidates, learned non-HDC reranking beats fixed fusion by +0.009121, CI [0.002248, 0.017206]. HDC is significantly worse at 50 and 100 candidates and has no demonstrated advantage at 200. Graph-answer top-1 also drops from 0.9785 to 0.9677.

Packed HDC uses 42,994,000 bytes at 8,000 dimensions. Its 200-candidate feature computation is fast (0.0674 ms/query on the recorded arm64 ReleaseFast run), but the feature does not improve relevance.

Machine-readable result:

• zig/bench/baselines/hdc-wands-candidate-reranker-2026-07-27.json

Full protocol, earlier exact/RaBitQ/packed experiments, lifecycle design, UX, references, and limitations:

• zig/HDC.md

Remaining production gates

This PR is a production-shaped hybrid execution prototype, not completion of the full product goal. Before advertising one stable cross-deployment contract, it still needs:

• a second external compositional or relational relevance workload;
• production latency, memory, fanout, response-size, and concurrency campaigns;
• provider timeout/failure and reranker concurrency policy;
• authorization/tenant and high-cardinality structured-signal pressure fixtures;
• SDK explanation rendering and pagination contract tests;
• serverless execution parity;
• HDC lifecycle/resource work only if later evidence reverses the current no-go.

Validation

• zig build root-test — 222 passed
• zig build serverless-test — 337 passed, 2 opt-in object-store integrations skipped
• zig build root-test -- --test-filter hybrid — 9 passed
• zig build root-test -- --test-filter fusion — 10 passed
• zig build api-table-reads-docid-test -- --test-filter reranks — 21 passed
• zig build hdc-wands-bench-test
• full pinned candidate-local WANDS experiment with 10,000-sample paired bootstrap
• make zig-generated-check
• git diff --check