# Feat: Add Codira Embedding Retrieval Quality Benchmark

[marco0560]

Feat: Add Codira Embedding Retrieval Quality Benchmark

Summary

Add a reproducible benchmark suite for measuring embedding model quality for Codira.

Current embedding campaigns measure latency, memory, backend overhead, and indexing cost well. They do not yet measure whether a given embedding model retrieves better context for Codira-specific development tasks.

This issue adds a versioned benchmark for retrieval quality, so Codira can decide whether larger or more expensive embedding models are actually worth their runtime and memory cost.

The benchmark should answer:

Given the same repository, commit, chunking configuration, backend, and query set, does model A retrieve more useful Codira context than model B?

It should not answer the vague question:

Is this embedding model better in general?

Codira should select embedding models based on measured retrieval quality under Codira workloads, not on vector dimensionality, model reputation, or generic leaderboard scores alone.

---

Motivation

Recent performance campaigns show that smaller models are materially cheaper than larger ones.

For example, the current measured default candidate remains:

```text
bge-small-en-v1.5-onnx + sqlite
```

because it has strong interactive ctx / emb performance, modest memory use, and avoids the cost of larger 768-dimensional models.

However, there are indications that some larger models may produce “better” semantic retrieval. That claim is currently not measured in a Codira-specific, reproducible way.

This benchmark should make the tradeoff explicit:

```text
quality gain vs latency cost vs memory cost vs indexing cost
```

A larger model should only become a default, or a recommended quality profile, if it produces a measurable retrieval-quality improvement large enough to justify its cost.

---

Non-goals

This issue should not initially attempt to:

• benchmark storage backends;
• make DuckDB the default benchmark backend;
• evaluate generic embedding leaderboard quality;
• integrate MTEB or BEIR directly;
• use an LLM judge during the main benchmark run;
• evaluate rerankers;
• evaluate split-engine indexing/query configurations;
• optimize chunking;
• compare vector databases.

Those can be separate follow-up issues.

The first benchmark should isolate embedding model retrieval quality as much as possible.

---

Background terminology

Codira does not need to expose all IR terminology to users, but the benchmark implementation should be compatible with standard ranked-retrieval metrics.

Useful concepts:

• Hit@K: whether at least one relevant result appears in the top K.
• Recall@K: how many known relevant results appear in the top K.
• MRR@K: how early the first relevant result appears.
• nDCG@K: ranking quality when relevance has graded scores, for example 3, 2, 1, 0.
• MAP: mean average precision; standard IR metric, but not required for the first implementation.
• MTEB: a broad embedding benchmark suite; useful context, but not a Codira-specific decision mechanism.
• BEIR: a retrieval benchmark suite; closer in spirit to Codira, but still not a substitute for Codira-specific evaluation.
• TREC-style pooling: collect top results from multiple systems/models, deduplicate them, and manually judge the pooled unique results once.

For the first implementation, Codira should focus on:

```text
Hit@1
Recall@5
MRR@5 or MRR@10
optional nDCG@10
```

---

Core design principle

Benchmark Codira retrieval tasks, not raw model quality.

A benchmark case is not just a query string. It is:

```text
repo + query + intent + expected relevant results
```

Different repositories should have different query text, but the same benchmark taxonomy.

Bad approach:

```text
Run the exact same query strings against every repository.
```

Good approach:

```text
Run the same categories of queries against every repository,
with repo-specific query text and repo-specific gold labels.
```

Example:

Intent	codira query	chatops query
symbol_lookup	DeclarationArtifact	ChatConversation
architecture_lookup	how does the backend plugin lifecycle work?	how does the processing pipeline work?
task_lookup	where would I change embedding vector persistence?	where would I change duplicate detection?
docs_lookup	how are analyzers configured?	how are ADRs created?

The benchmark should compare models on equivalent task types, not on identical query strings.

---

Initial benchmark scope

Use a small but structured first suite.

Recommended first version:

```text
4 repositories
5 query categories
5 queries per category per repository
= 100 benchmark cases
```

Suggested initial repositories:

```text
codira
chatops
fontshow
sanikey
```

The exact list should come from the existing benchmark repository manifest where possible.

The benchmark should use fixed commits. If a repository changes during a campaign, the report must flag it and either exclude it from before/after conclusions or report it separately.

---

Query taxonomy

Each benchmark case should have an intent.

Initial intent categories:

```text
symbol_lookup
architecture_lookup
task_lookup
docs_lookup
error_or_trace_lookup
```

1. symbol_lookup

Queries that should retrieve a specific class, function, method, module, or declaration.

Examples:

```text
DeclarationArtifact
ChatConversation
EmbeddingModelConfig
DuckDBVectorStore
```

These cases can often be generated semi-automatically from the symbol table.

2. architecture_lookup

Natural-language queries about where an architectural concept is implemented or documented.

Examples:

```text
how does codira select the analyzer for a file?
how does the backend plugin lifecycle work?
how does the processing pipeline work?
where is stable conversation fingerprinting defined?
```

3. task_lookup

Queries phrased as development tasks.

Examples:

```text
where would I change how pending embedding rows are flushed?
what code would I touch to add markdown frontmatter support?
where would I change duplicate detection?
where is keyboard layout data generated?
```

4. docs_lookup

Queries where the relevant answer may be in documentation rather than code.

Examples:

```text
how are analyzers configured?
how are ADRs created?
how is the embedding backend configured?
what is the documented storage backend policy?
```

5. error_or_trace_lookup

Queries based on error messages, failure symptoms, or stack trace fragments.

Examples:

```text
PRIMARY KEY or UNIQUE constraint violation duplicate key
unrecognized arguments benchmarks/bk-cpp.local.json
EXPORT_SCHEMA_ERROR missing role
DuckDB executemany semgrep finding
```

---

Query distribution

For the first version:

Repo	Symbol	Architecture	Task	Docs	Error/trace	Total
codira	5	5	5	5	5	25
chatops	5	5	5	5	5	25
fontshow	5	5	5	5	5	25
sanikey	5	5	5	5	5	25
Total	20	20	20	20	20	100

A later mature version can expand to:

```text
10 queries × 5 categories × 4 repositories = 200 cases
```

or eventually:

```text
300+ cases
```

but the first version should not require 300 hand-graded queries.

---

Manual grading strategy

Do not manually grade every model result independently.

Use TREC-style pooling:

1. Run every candidate model on the same benchmark cases.
2. For each query, collect the top K results from all models.
3. Deduplicate the result set.
4. Manually grade each unique result once.
5. Reuse those labels to score every model.

Example:

```text
Query: "where is embedding flush implemented?"

Model A top 5:
A, B, C, D, E

Model B top 5:
A, C, F, G, H

Model C top 5:
B, F, I, J, K

Pooled unique results:
A, B, C, D, E, F, G, H, I, J, K
```

The grader judges 11 unique results, not 15 model-specific rows.

This keeps the benchmark feasible.

---

Grading scale

The first implementation should support both binary and graded relevance.

Binary mode

Recommended for the first suite:

```text
1 = useful
0 = not useful
```

This is enough for:

```text
Hit@1
Recall@5
MRR@5
MRR@10
```

Graded mode

Optional, for later refinement:

```text
3 = ideal / central result
2 = useful result
1 = marginally useful result
0 = irrelevant result
```

This enables:

```text
nDCG@5
nDCG@10
```

The benchmark data format should allow graded labels from the start, even if the first dataset uses only 0 and 1.

---

Benchmark case format

Add a JSONL file such as:

```text
benchmarks/embedding-quality/cases.jsonl
```

Example record:

```json
{
"id": "codira-task-embeddings-001",
"repo": "codira",
"intent": "task_lookup",
"language": "python",
"channel": "code",
"difficulty": "medium",
"query": "where would I change how pending embedding rows are flushed?",
"relevant": [
{
"kind": "symbol",
"id": "embeddings.flush_pending_rows",
"grade": 3
},
{
"kind": "file",
"path": "packages/codira-backend-duckdb/src/codira_backend_duckdb/...",
"grade": 2
}
]
}
```

Fields:

Field	Required	Description
id	yes	Stable benchmark case ID.
repo	yes	Repository key from the repo manifest.
intent	yes	Query category.
language	recommended	Main expected language/domain.
channel	recommended	code, docs, code+docs, tests, config, etc.
difficulty	recommended	easy, medium, hard.
query	yes	Natural query text passed to Codira.
relevant	yes	Known relevant files/symbols/docs with grades.

Relevant target kinds should support at least:

```text
file
symbol
doc_section
test
config
```

---

Gold-label format

The first implementation can keep labels directly in cases.jsonl.

A later implementation may split them:

```text
benchmarks/embedding-quality/cases.jsonl
benchmarks/embedding-quality/labels.jsonl
benchmarks/embedding-quality/pools.jsonl
```

Possible pooled-label record:

```json
{
"case_id": "codira-task-embeddings-001",
"result_id": "file:packages/codira-backend-duckdb/src/...",
"grade": 2,
"graded_by": "manual",
"notes": "Useful implementation file, but not the main flush function."
}
```

---

Campaign command

Add a script or CLI command similar to:

```bash
uv run python -m scripts.run_embedding_quality_campaign
--manifest benchmarks/uv-backed-repos.local.json
--model-manifest benchmarks/embedding-model-candidates.json
--queries benchmarks/embedding-quality/cases.jsonl
--backend sqlite
--k 1,3,5,10,20
--output .artifacts/embedding-quality-campaign/$(date -u +%Y%m%dT%H%M%SZ)
```

The first version should use:

```text
backend: sqlite
```

Rationale:

• the benchmark is about embedding retrieval quality;
• SQLite is currently the stable production backend;
• DuckDB still has known full-index write-path overhead;
• backend comparison is a separate concern.

---

Candidate models

The benchmark should use the existing embedding model manifest where possible:

```text
benchmarks/embedding-model-candidates.json
```

It should compare at least:

```text
current default candidate
best 384-dimensional ONNX candidate
one or more 768-dimensional ONNX candidates
sentence-transformers baseline if still relevant
```

The benchmark should report model metadata:

```text
model name
engine
dimension
backend
index latency
query latency
memory/RSS
quality metrics
```

---

Metrics

Compute at least:

```text
Hit@1
Recall@5
MRR@5
MRR@10
```

Optionally:

```text
Recall@10
nDCG@5
nDCG@10
MAP
```

The first report should not depend on MAP.

MAP can be added later if useful.

---

Aggregation strategy

Do not report only one global number.

Report:

1. Per-repository metrics

Example:

Model	Repo	Hit@1	Recall@5	MRR@10	nDCG@10
bge-small-en-v1.5-onnx	codira	...	...	...	...
bge-small-en-v1.5-onnx	chatops	...	...	...	...

This shows whether a model only wins on one repository.

2. Per-intent metrics

Example:

Model	Intent	Hit@1	Recall@5	MRR@10	nDCG@10
bge-small-en-v1.5-onnx	symbol_lookup	...	...	...	...
bge-small-en-v1.5-onnx	architecture_lookup	...	...	...	...
bge-small-en-v1.5-onnx	task_lookup	...	...	...	...
bge-small-en-v1.5-onnx	docs_lookup	...	...	...	...

This is likely more important than the global score.

3. Macro average

Each repository gets equal weight.

```text
macro_avg = mean(score_per_repo)
```

This prevents a larger repository or a repository with more cases from dominating the final decision.

4. Micro average

Each query gets equal weight.

```text
micro_avg = total successful results / total evaluated cases
```

This is useful as a secondary number.

The report should clearly label macro and micro averages.

---

Decision gates

A larger or slower model should only be recommended if it shows a material quality gain.

Initial suggested gate:

```text
A more expensive model may become the default only if:

• Recall@5 improves by at least +5 percentage points, or
• MRR@10 improves by at least +0.05 absolute, or
• nDCG@10 improves by at least +0.05 absolute,
  while:
• ctx latency does not regress beyond 1.25x,
• emb latency does not regress beyond 1.25x,
• max RSS remains within the configured memory budget,
• full-index cost remains acceptable for the target profile.
  ```

If a larger model improves quality but violates default latency/memory constraints, it may be classified as:

```text
quality profile
```

instead of:

```text
default profile
```

Possible decisions:

Outcome	Meaning
default	Best balance of retrieval quality, latency, memory, and index cost.
quality_profile	Better retrieval quality, but too expensive for default use.
interactive_profile	Best for ctx / emb latency, acceptable quality.
rejected	Cost increase is not justified by retrieval improvement.
needs_more_data	Difference is too small or inconsistent across repos/intents.

---

Output artifacts

The campaign should write:

```text
quality-summary.md
quality-summary.json
per-query-results.jsonl
per-model-ranking.csv
per-repo-summary.csv
per-intent-summary.csv
pooled-results.jsonl
grading-sheet.csv
regressions.md
```

quality-summary.md

Human-readable report with:

• campaign metadata;
• repo commit parity;
• model manifest;
• backend;
• query set version;
• per-model summary;
• per-repo metrics;
• per-intent metrics;
• macro average;
• micro average;
• latency and memory comparison;
• final recommendation.

quality-summary.json

Machine-readable summary for later comparison.

per-query-results.jsonl

One row per model/query/result.

Should include:

```text
case_id
repo
intent
model
rank
result kind
result path/symbol
score
grade
is_relevant
```

pooled-results.jsonl

Deduplicated results requiring or containing manual labels.

grading-sheet.csv

A human-editable spreadsheet-style file for manual grading.

regressions.md

Highlights cases where a model:

• lost a previously relevant top result;
• moved the first relevant result lower;
• failed an easy symbol lookup;
• performed poorly on a specific repo or intent.

---

Reproducibility requirements

Each campaign must record:

```text
Codira version
git commit of Codira
repo manifest path
model manifest path
query set path
query set checksum
backend
chunking configuration
embedding configuration
K values
timestamp
host platform
Python version
ONNX / sentence-transformers versions where available
repository commits
```

The report must flag:

• repository commit drift;
• missing repositories;
• failed indexing;
• failed queries;
• model failures;
• empty result sets;
• incompatible dimensions/configurations;
• missing gold labels.

---

Implementation sketch

Phase 1: Data format and validation

Add:

```text
benchmarks/embedding-quality/cases.jsonl
benchmarks/embedding-quality/README.md
benchmarks/embedding-quality/grading-guidelines.md
```

Add validation for:

• unique case IDs;
• known repositories;
• known intents;
• non-empty query text;
• valid relevance grades;
• valid target kinds;
• valid paths/symbol IDs where checkable.

Phase 2: Campaign runner

Add:

```text
scripts/run_embedding_quality_campaign.py
```

Responsibilities:

1. Load repository manifest.
2. Load model manifest.
3. Load benchmark cases.
4. For each model/repo pair:
   • build or reuse index;
   • run retrieval command;
   • collect top K results.
5. Write raw per-query results.
6. Build pooled result set.
7. If labels exist, compute metrics.
8. Write summary artifacts.

Phase 3: Metric engine

Add a small metric module supporting:

```text
Hit@K
Recall@K
MRR@K
nDCG@K
macro average
micro average
per-repo aggregation
per-intent aggregation
```

Keep this module independent from the campaign runner where practical.

Phase 4: Manual grading workflow

Generate:

```text
grading-sheet.csv
```

with columns such as:

```text
case_id
repo
intent
query
result_id
ranked_by_models
result_kind
result_path
result_symbol
grade
notes
```

Then allow re-running the metric computation after the grading sheet has been filled.

Possible command:

```bash
uv run python -m scripts.score_embedding_quality_campaign
--results .artifacts/embedding-quality-campaign//per-query-results.jsonl
--labels benchmarks/embedding-quality/labels.jsonl
--output .artifacts/embedding-quality-campaign//scored
```

Phase 5: Reporting

Generate Markdown and JSON summaries.

The Markdown report should include the decision table:

Model	Dim	Engine	Backend	Macro Recall@5	Macro MRR@10	ctx median	emb median	Max RSS	Decision
bge-small-en-v1.5-onnx	384	ONNX	sqlite	...	...	...	...	...	default
jina-embeddings-v2-base-code-onnx	768	ONNX	sqlite	...	...	...	...	...	quality_profile
nomic-embed-text-v1.5-onnx	768	ONNX	sqlite	...	...	...	...	...	rejected

---

Suggested first benchmark workflow

1. Create 100 benchmark cases:

   • 4 repos;
   • 5 intents;
   • 5 queries per intent per repo.

2. Run candidate models with top_k = 10.

3. Pool results across models.

4. Manually grade pooled unique results using binary relevance:

```text
1 = useful
0 = not useful
```

5. Compute:

```text
Hit@1
Recall@5
MRR@5
MRR@10
```

6. Compare models using:

   • per-repo metrics;
   • per-intent metrics;
   • macro average;
   • micro average;
   • latency;
   • memory;
   • full-index cost.

7. Only expand to 200+ cases if the first 100 cases show a meaningful difference.

---

Testing requirements

Add tests for:

• loading valid benchmark cases;
• rejecting duplicate case IDs;
• rejecting invalid intent names;
• rejecting invalid relevance grades;
• computing Hit@K;
• computing Recall@K;
• computing MRR@K;
• computing nDCG@K if implemented;
• macro aggregation;
• micro aggregation;
• pooled result deduplication;
• stable output ordering;
• report generation with minimal fixture data;
• campaign failure reporting when a model or repo fails.

Use small fixtures, not real large repositories, for unit tests.

---

Documentation requirements

Add documentation explaining:

```text
docs/benchmarks/embedding-quality.md
```

The document should cover:

• why Codira needs a quality benchmark;
• why generic embedding leaderboards are insufficient;
• benchmark case format;
• query taxonomy;
• grading guidelines;
• pooling workflow;
• metrics;
• macro vs micro averages;
• decision gates;
• how to run the campaign;
• how to update labels;
• how to interpret results.

Also update any relevant benchmark/process index pages.

---

Acceptance criteria

This issue is complete when:

• Codira has a versioned embedding quality benchmark case format.
• At least one initial cases.jsonl file exists.
• The benchmark supports repo-specific queries with a shared intent taxonomy.
• The runner can evaluate multiple embedding models against the same case set.
• Results are collected at configurable K values.
• Pooled results can be generated for manual grading.
• Binary relevance labels are supported.
• Graded relevance labels are supported or the format is forward-compatible with them.
• Hit@1 is computed.
• Recall@5 is computed.
• MRR@5 or MRR@10 is computed.
• Per-repo summaries are generated.
• Per-intent summaries are generated.
• Macro averages are generated.
• Micro averages are generated.
• Markdown and JSON reports are emitted.
• Campaign metadata records repo commits, Codira version, model manifest, query set checksum, backend, and chunking/embedding configuration.
• The first benchmark defaults to SQLite to avoid mixing retrieval-quality measurement with DuckDB write-path overhead.
• Documentation explains how to create, grade, run, and interpret the benchmark.
• Tests cover data validation, metric computation, pooling, aggregation, and report generation.

---

Future extensions

Possible follow-up issues:

• Add a larger 200–300 case benchmark set.
• Add nDCG-based graded evaluation as the default once enough labels exist.
• Add MAP if useful.
• Add automatic query generation from symbols, docs headings, tests, and issue metadata.
• Add regression tracking between campaigns.
• Add support for split-engine evaluation, for example Torch indexing plus ONNX query.
• Add reranker evaluation.
• Add channel-specific scoring, for example code, docs, tests, config.
• Add language-specific scoring.
• Add automatic comparison against previous quality campaigns.
• Add CI smoke tests with a tiny fixture repo.
• Add a quality profile recommendation mechanism in the embedding model manifest.

---

Design note

Codira should not select embedding models based on vector dimension or generic benchmark reputation.

Codira should select embedding models based on:

```text
retrieval quality on versioned Codira benchmark cases
under explicit latency, memory, and indexing-cost constraints
```

A 768-dimensional model is only better for Codira if it retrieves better Codira context by enough margin to justify its cost.