diff --git a/docs/research/memory-formation-query-transform/06-candidate-depth-diversity-results.md b/docs/research/memory-formation-query-transform/06-candidate-depth-diversity-results.md new file mode 100644 index 0000000..3d98504 --- /dev/null +++ b/docs/research/memory-formation-query-transform/06-candidate-depth-diversity-results.md @@ -0,0 +1,107 @@ +# Candidate Depth and Near-Duplicate Diversity Ablation + +Date: 2026-08-23 + +Status: research result; no deployment or production data changes + +## Question + +The lexical coverage floor raised the noisy-prefix corpus to 14/20 gold at +rank one. All six remaining gold memories were at 0-based pool rank 10: the +first item excluded from a final top-10 page. Each was preceded by ten +almost-identical archived neighboring-service notes. + +This ablation separated two hypotheses: + +1. A deeper candidate pool alone lets the existing ranker resolve the cluster. +2. The result page needs an explicit diversity constraint so one + near-duplicate cluster cannot consume every slot. + +## Controls + +- Base commit: `47e8c82` +- Required flags in every arm: `RECALL_RERANK_SCALE_FIX`, + `RECALL_LEXICAL_COVERAGE_FLOOR`, and `RECALL_RESCUE_SQL_TIEBREAK` +- Final caller-visible top-K: 10 +- Candidate depths: 10, 12, 20, 50 +- Existing 20-task `mnemon-v02-noise-prefix` receipt and tasks +- Usage counters restored from `zz_probe_usage_snapshot` before every page and + deep-pool query, preventing cross-query usage-weight feedback +- Two independent server restarts/runs per arm; no LLM generation + +An initial pilot reset usage only once per arm. It exposed a harness confound: +arms returning different memories produced different usage state for later +queries. Those pilot numbers are excluded below. + +## Depth-only result (negative) + +| Candidate depth | gold@1 | gold@5 | gold@10 | MRR@10 | Former misses | +| ---: | ---: | ---: | ---: | ---: | --- | +| 10 | 14/20 | 14/20 | 14/20 | 0.700 | absent | +| 12 | 14/20 | 14/20 | 14/20 | 0.700 | pool rank 10 | +| 20 | 14/20 | 14/20 | 14/20 | 0.700 | pool rank 10 | +| 50 | 14/20 | 14/20 | 14/20 | 0.700 | pool rank 10 | + +Depth makes gold available but does not change final ordering. More candidates +alone buy zero retrieval quality here. + +## Diversity intervention + +The research prototype greedily preserves relevance order while allowing at +most two members from any token-set Jaccard cluster at similarity >= 0.90. +Suppressed rows are then backfilled in original relevance order so the API +still returns a complete top-10 page. It does not delete or mutate memories. + +The candidate controls explicitly depend on `RECALL_RERANK_SCALE_FIX=true`: +diversifying a mixture of raw rescue-band and post-rerank scores is undefined, +so both controls are ignored when the scale fix is off. Configured candidate +depth is a **minimum**, not a maximum: the effective depth is +`max(caller limit, configured depth)`, so a caller asking for more than 12 rows +is never truncated. If the cluster cap is enabled without an explicit depth, +the service uses a bounded 50-row scan (or the caller limit when larger). + +The threshold is conservative for this failure mode: sampled decoy-to-decoy +similarity is 0.951 while gold-to-decoy similarity is 0.178. The mechanism +does not inspect benchmark IDs, `archived` wording, or gold labels. + +| Candidate depth | cluster cap | gold@1 | gold@5 | gold@10 | MRR@10 | Page size | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 12 | 2 | 14/20 | **20/20** | **20/20** | **0.800** | 10 | +| 20 | 2 | 14/20 | **20/20** | **20/20** | **0.800** | 10 | + +At the smallest winning depth (12), all six former misses moved from 0-based +pool rank 10 to page rank 2. No former hit regressed. Both repeats were +bit-identical: 0/20 query rankings changed. + +The wider matrix tested caps 4 and 8. Cap 4 moved all former misses to rank 4 +(20/20 gold@5, MRR 0.760). Cap 8 moved them to rank 8 (20/20 gold@10, MRR +0.7333). Cap 2 is strongest while retaining two cluster representatives. + +## Latency + +- depth-10 control: p50 39-43 ms; p95 69-73 ms +- depth-12 + cap-2: p50 30-42 ms; p95 77-83 ms + +There is no demonstrated material latency cost at this scale. A larger corpus +and production load test are still required before making a latency claim. + +## Recommendation + +The smallest winner is candidate depth 12, final top-K 10, Jaccard threshold +0.90, and at most two initially selected rows per cluster followed by +relevance-order backfill. + +Keep it feature-flagged. It improves gold@5 from 14/20 to 20/20 and MRR@10 from +0.700 to 0.800 with no observed regression, no model call, and stable repeats. + +Do not ship from this benchmark alone. Boilerplate-heavy decoys are deliberate, +and near-identical text can encode real version/environment distinctions. Next +validate ordinary noise, stale/superseded facts, contradictions, ambiguous +cross-project facts, false-positive injection, and downstream executable +quality. + +## Artifacts + +Raw JSON, server logs, repeats, former-miss ranks, page sizes, and latency +summaries are under `artifacts/candidate-matrix/`. Final backfill validation is +in `isolated-backfill/`; the wider isolated sweep is in `isolated/`. diff --git a/docs/research/memory-formation-query-transform/07-diversity-production-gate.md b/docs/research/memory-formation-query-transform/07-diversity-production-gate.md new file mode 100644 index 0000000..2c1cca4 --- /dev/null +++ b/docs/research/memory-formation-query-transform/07-diversity-production-gate.md @@ -0,0 +1,113 @@ +# Candidate-diversity production gate + +Date: 2026-08-23 + +This gate independently validates commit `08be7a4` against the 81-query, +1,210-memory real-embedding benchmark. It compares the combined recall fixes +with candidate diversity disabled and enabled. Nothing was pushed, deployed, or +run against production data. + +## Configuration + +Both arms enabled: + +- `RECALL_RERANK_SCALE_FIX=true` +- `RECALL_LEXICAL_COVERAGE_FLOOR=true` +- `RECALL_RESCUE_SQL_TIEBREAK=true` +- final result limit 10 + +The treatment additionally enabled: + +- `RECALL_CANDIDATE_POOL_DEPTH=12` +- `RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT=2` +- token-set Jaccard threshold 0.90 + +The corpus used real `bge-base` 768-dimensional embeddings in a dedicated local +database. The raw artifact is +[`artifacts/diversity-production-gate/real-embedding-81.json`](artifacts/diversity-production-gate/real-embedding-81.json). + +## Benchmark validity controls + +Issue #326 identified two confounds in the existing harness. This run controls +both: + +1. `seedCorpus` silently drops fixture tags, metadata, and memory type. The gate + restored all declared values, then compared the stored values with the fixture + definitions for all 1,210 memories. All 1,210 memories had their declared tags. +2. Recall mutates usage fields that can change later rankings. The gate took a + persistent snapshot of `retrieval_count`, `used_count`, `unused_count`, + `last_retrieved_at`, and `last_used_at`, then restored it before every query + and every repeat. + +Every query was executed twice in each arm. No query changed ranking between +identical repeats in either arm. + +## Results + +| metric | diversity off | diversity on | delta | +| --- | ---: | ---: | ---: | +| required hit rate@5 | 60/62 (96.77%) | 60/62 (96.77%) | 0 | +| relevant recall@10 | 71/73 (97.26%) | 71/73 (97.26%) | 0 | +| MRR@10 | 0.91262 | 0.91282 | +0.00020 | +| forbidden-memory injection | 0/91 | 0/91 | 0 | +| full-page rate | 73/80 (91.25%) | 73/80 (91.25%) | 0 | +| mean page size | 9.4375 | 9.4375 | 0 | +| latency p50 | 23 ms | 18 ms | -5 ms | +| latency p95 | 31 ms | 38 ms | +7 ms | +| latency mean | 21.83 ms | 21.46 ms | -0.36 ms | + +There were no top-five gains and no top-five losses. Eight emotional-query +rankings changed. Only `emotional_003` changed a relevant rank: the second +required memory moved from rank 9 to rank 7, producing the small MRR increase. +The other seven changes reordered non-required results. Page sizes were +identical query by query. + +The latency sample is an execution sanity check, not a performance claim. Arms +ran sequentially rather than in an interleaved latency experiment; the mixed +p50/p95 movement does not establish a speedup or regression. + +## Relationship to the targeted near-duplicate result + +The controlled mnemon near-duplicate matrix remains the positive efficacy +evidence. Depth alone did not move the 14/20 ceiling. Depth 12 plus a two-member +near-duplicate cluster cap retained 14/20 at rank 1 and moved all six remaining +gold memories into the returned page: 20/20 hit@5, 20/20 hit@10, and MRR 0.800. +Two independent repeats were exact. + +The 81-query gate supplies the complementary safety evidence: the same setting +did not trade away required top-five hits, recall, tenant isolation, stability, +or page fullness on the ordinary real-embedding suite. + +## Graduated-corpus gap + +Phase 1 defines deterministic generators for the requested conditions in +mnemon: + +- G1: 20 targets plus 400 ordinary unrelated memories +- G3: 20 targets plus 200 stale/superseded memories +- G4: 20 targets plus 100 directly contradictory memories +- G5: 20 targets plus 200 ambiguous cross-project memories + +No materialized corpus with a successful embedding-gate receipt or completed +retrieval report exists for these conditions. The only attempted smoke run was +invalid: it produced no usable embeddings because the local embedding path +returned NaNs and mixed 384-dimensional output with the 1,536-dimensional test +configuration. The seeder cleaned that attempt up. These conditions therefore +have **no result**, not a passing or failing result. + +The 81-query suite contains temporal, emotional, adversarial, cross-feature, +edge-case, and tenant-isolation queries, but it is not a causal substitute for +G1/G3/G4/G5. + +## Verdict + +Commit `08be7a4` earns a merge recommendation **behind its existing default-off +flags**. The targeted near-duplicate condition shows a large, replicated gain, +and the corrected 81-query real-embedding gate finds no quality, isolation, +stability, or page-fullness regression. + +It does not yet earn default-on production rollout. Before changing defaults, +materialize and embedding-gate G1, G3, G4, and G5; run the same isolated on/off +comparison; then run an interleaved latency/load check. A small internal canary +with decision telemetry is reasonable after merge because rollback is a flag +change. diff --git a/docs/research/memory-formation-query-transform/README.md b/docs/research/memory-formation-query-transform/README.md index 83de48b..400c6ea 100644 --- a/docs/research/memory-formation-query-transform/README.md +++ b/docs/research/memory-formation-query-transform/README.md @@ -20,6 +20,9 @@ interventions in Engram's memory lifecycle: | 2 | [`02-research-memo.md`](02-research-memo.md) | Literature review (formation + query-transform), blind-spot pass, promising-vs-hype read, cited primary sources | | 3 | [`03-experiment-spec.md`](03-experiment-spec.md) | A–E ablation matrix, graduated corpora, retrieval + downstream metrics, reproducibility controls, budgets | | 4 | [`04-finding-tie-domination.md`](04-finding-tie-domination.md) | **Blocking finding.** Retrieval ranking is decided by array order among tied scores, not by score. Changes the required sequencing. | +| 5 | [`05-finding-band-inversion.md`](05-finding-band-inversion.md) | Rerank-scale mixing finding and scale-fix ablation. | +| 6 | [`06-candidate-depth-diversity-results.md`](06-candidate-depth-diversity-results.md) | Candidate-depth control and near-duplicate diversity ablation that breaks the 14/20 ceiling. | +| 7 | [`07-diversity-production-gate.md`](07-diversity-production-gate.md) | Corrected 81-query real-embedding safety gate for candidate diversity, including issue #326 controls and graduated-corpus gaps. | ## Executive summary diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/final-summary.txt b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/final-summary.txt new file mode 100644 index 0000000..6e31d87 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/final-summary.txt @@ -0,0 +1,92 @@ +┌─────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────┬────┬─────────┬─────────┬──────────┬─────────┬───────────┬───────────┬──────────────┬──────────────┐ +│ (index) │ artifact │ arm │ n │ goldAt1 │ goldAt5 │ goldAt10 │ mrrAt10 │ pageSizes │ poolSizes │ latencyP50Ms │ latencyP95Ms │ +├─────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┼────┼─────────┼─────────┼──────────┼─────────┼───────────┼───────────┼──────────────┼──────────────┤ +│ 0 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth10-r1.json' │ 'depth10-r1' │ 20 │ 14 │ 14 │ 14 │ 0.7 │ '10' │ '10' │ 39 │ 73 │ +│ 1 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth10-r2.json' │ 'depth10-r2' │ 20 │ 14 │ 14 │ 14 │ 0.7 │ '10' │ '10' │ 43 │ 69 │ +│ 2 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth12-r1.json' │ 'depth12-r1' │ 20 │ 14 │ 14 │ 14 │ 0.7 │ '10' │ '12' │ 43 │ 72 │ +│ 3 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth12-r2.json' │ 'depth12-r2' │ 20 │ 14 │ 14 │ 14 │ 0.7 │ '10' │ '12' │ 42 │ 70 │ +│ 4 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth20-r1.json' │ 'depth20-r1' │ 20 │ 14 │ 14 │ 14 │ 0.7 │ '10' │ '20' │ 42 │ 69 │ +│ 5 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth20-r2.json' │ 'depth20-r2' │ 20 │ 14 │ 14 │ 14 │ 0.7 │ '10' │ '20' │ 41 │ 69 │ +│ 6 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth50-r1.json' │ 'depth50-r1' │ 20 │ 14 │ 14 │ 14 │ 0.7 │ '10' │ '50' │ 42 │ 66 │ +│ 7 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth50-r2.json' │ 'depth50-r2' │ 20 │ 14 │ 14 │ 14 │ 0.7 │ '10' │ '50' │ 41 │ 67 │ +│ 8 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12-r1.json' │ 'cluster12-r1' │ 20 │ 14 │ 20 │ 20 │ 0.8 │ '10' │ '12' │ 42 │ 77 │ +│ 9 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12-r2.json' │ 'cluster12-r2' │ 20 │ 14 │ 20 │ 20 │ 0.8 │ '10' │ '12' │ 30 │ 83 │ +│ 10 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster20-r1.json' │ 'cluster20-r1' │ 20 │ 14 │ 20 │ 20 │ 0.8 │ '10' │ '20' │ 44 │ 77 │ +│ 11 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster20-r2.json' │ 'cluster20-r2' │ 20 │ 14 │ 20 │ 20 │ 0.8 │ '10' │ '20' │ 44 │ 68 │ +│ 12 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c4-r1.json' │ 'cluster12c4-r1' │ 20 │ 14 │ 20 │ 20 │ 0.76 │ '10' │ '12' │ 43 │ 87 │ +│ 13 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c4-r2.json' │ 'cluster12c4-r2' │ 20 │ 14 │ 20 │ 20 │ 0.76 │ '10' │ '12' │ 45 │ 73 │ +│ 14 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c8-r1.json' │ 'cluster12c8-r1' │ 20 │ 14 │ 14 │ 20 │ 0.7333 │ '10' │ '12' │ 45 │ 73 │ +│ 15 │ 'docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c8-r2.json' │ 'cluster12c8-r2' │ 20 │ 14 │ 14 │ 20 │ 0.7333 │ '10' │ '12' │ 45 │ 80 │ +└─────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴────┴─────────┴─────────┴──────────┴─────────┴───────────┴───────────┴──────────────┴──────────────┘ +depth10 stability: 0/20 repeated-query rankings changed +depth12 stability: 0/20 repeated-query rankings changed +depth20 stability: 0/20 repeated-query rankings changed +depth50 stability: 0/20 repeated-query rankings changed +cluster12 stability: 0/20 repeated-query rankings changed +cluster20 stability: 0/20 repeated-query rankings changed +cluster12c4 stability: 0/20 repeated-query rankings changed +cluster12c8 stability: 0/20 repeated-query rankings changed + +depth10-r1 former-miss ranks (0-based): + paginated-list-endpoint: page=-1 pool=-1 poolSize=10 + webhook-processor: page=-1 pool=-1 poolSize=10 + inventory-reservation: page=-1 pool=-1 poolSize=10 + currency-conversion: page=-1 pool=-1 poolSize=10 + tenant-scoped-projects: page=-1 pool=-1 poolSize=10 + order-dto-mapping: page=-1 pool=-1 poolSize=10 + +depth12-r1 former-miss ranks (0-based): + paginated-list-endpoint: page=-1 pool=10 poolSize=12 + webhook-processor: page=-1 pool=10 poolSize=12 + inventory-reservation: page=-1 pool=10 poolSize=12 + currency-conversion: page=-1 pool=10 poolSize=12 + tenant-scoped-projects: page=-1 pool=10 poolSize=12 + order-dto-mapping: page=-1 pool=10 poolSize=12 + +depth20-r1 former-miss ranks (0-based): + paginated-list-endpoint: page=-1 pool=10 poolSize=20 + webhook-processor: page=-1 pool=10 poolSize=20 + inventory-reservation: page=-1 pool=10 poolSize=20 + currency-conversion: page=-1 pool=10 poolSize=20 + tenant-scoped-projects: page=-1 pool=10 poolSize=20 + order-dto-mapping: page=-1 pool=10 poolSize=20 + +depth50-r1 former-miss ranks (0-based): + paginated-list-endpoint: page=-1 pool=10 poolSize=50 + webhook-processor: page=-1 pool=10 poolSize=50 + inventory-reservation: page=-1 pool=10 poolSize=50 + currency-conversion: page=-1 pool=10 poolSize=50 + tenant-scoped-projects: page=-1 pool=10 poolSize=50 + order-dto-mapping: page=-1 pool=10 poolSize=50 + +cluster12-r1 former-miss ranks (0-based): + paginated-list-endpoint: page=2 pool=2 poolSize=12 + webhook-processor: page=2 pool=2 poolSize=12 + inventory-reservation: page=2 pool=2 poolSize=12 + currency-conversion: page=2 pool=2 poolSize=12 + tenant-scoped-projects: page=2 pool=2 poolSize=12 + order-dto-mapping: page=2 pool=2 poolSize=12 + +cluster20-r1 former-miss ranks (0-based): + paginated-list-endpoint: page=2 pool=2 poolSize=20 + webhook-processor: page=2 pool=2 poolSize=20 + inventory-reservation: page=2 pool=2 poolSize=20 + currency-conversion: page=2 pool=2 poolSize=20 + tenant-scoped-projects: page=2 pool=2 poolSize=20 + order-dto-mapping: page=2 pool=2 poolSize=20 + +cluster12c4-r1 former-miss ranks (0-based): + paginated-list-endpoint: page=4 pool=4 poolSize=12 + webhook-processor: page=4 pool=4 poolSize=12 + inventory-reservation: page=4 pool=4 poolSize=12 + currency-conversion: page=4 pool=4 poolSize=12 + tenant-scoped-projects: page=4 pool=4 poolSize=12 + order-dto-mapping: page=4 pool=4 poolSize=12 + +cluster12c8-r1 former-miss ranks (0-based): + paginated-list-endpoint: page=8 pool=8 poolSize=12 + webhook-processor: page=8 pool=8 poolSize=12 + inventory-reservation: page=8 pool=8 poolSize=12 + currency-conversion: page=8 pool=8 poolSize=12 + tenant-scoped-projects: page=8 pool=8 poolSize=12 + order-dto-mapping: page=8 pool=8 poolSize=12 diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12-r1.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12-r1.json new file mode 100644 index 0000000..d64360e --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12-r1.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "cluster12-r1", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "20/20", + "goldTop10": "20/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 77, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 48, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33814625000000004, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 40, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jag00518kc97v7aiok2", + "score": 0.3899548569950754, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 43, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4jws00938kc9l5d2ca5p", + "score": 0.39057304910160945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 47, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 40, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mjz00p88kc9jgz2p1n4", + "score": 0.39016156237845945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 49, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 39, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4p4s01468kc93dp1x5it", + "score": 0.38992495978753283, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3413975, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 38, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.33770472222222225, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 42, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rpe01jn8kc9hf03r7k1", + "score": 0.39035138596571384, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 39, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 38, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 44, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4uau01yq8kc9ara0a13p", + "score": 0.39050281163664174, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: i" + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12-r2.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12-r2.json new file mode 100644 index 0000000..3d612cf --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12-r2.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "cluster12-r2", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "20/20", + "goldTop10": "20/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 83, + "deepLatencyMs": 54, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 49, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33814625000000004, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jag00518kc97v7aiok2", + "score": 0.3899548569950754, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 43, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4jws00938kc9l5d2ca5p", + "score": 0.39057304910160945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 49, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 28, + "deepLatencyMs": 25, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 25, + "deepLatencyMs": 25, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 23, + "deepLatencyMs": 23, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mjz00p88kc9jgz2p1n4", + "score": 0.39016156237845945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 24, + "deepLatencyMs": 33, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 32, + "deepLatencyMs": 25, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 24, + "deepLatencyMs": 24, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 30, + "deepLatencyMs": 24, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4p4s01468kc93dp1x5it", + "score": 0.38992495978753283, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3413975, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 30, + "deepLatencyMs": 28, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 24, + "deepLatencyMs": 28, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 23, + "deepLatencyMs": 36, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.33770472222222225, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 32, + "deepLatencyMs": 42, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rpe01jn8kc9hf03r7k1", + "score": 0.39035138596571384, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 27, + "deepLatencyMs": 36, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 29, + "deepLatencyMs": 28, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 35, + "deepLatencyMs": 35, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 37, + "deepLatencyMs": 39, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4uau01yq8kc9ara0a13p", + "score": 0.39050281163664174, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: i" + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c4-r1.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c4-r1.json new file mode 100644 index 0000000..4ac9493 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c4-r1.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "cluster12c4-r1", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "20/20", + "goldTop10": "20/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 87, + "deepLatencyMs": 58, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 46, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33814625000000004, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 41, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jag00518kc97v7aiok2", + "score": 0.3899548569950754, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 53, + "deepLatencyMs": 43, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4jws00938kc9l5d2ca5p", + "score": 0.39057304910160945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 46, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 39, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 38, + "deepLatencyMs": 40, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mjz00p88kc9jgz2p1n4", + "score": 0.39016156237845945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 41, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4p4s01468kc93dp1x5it", + "score": 0.38992495978753283, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3413975, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.33770472222222225, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 44, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4rpe01jn8kc9hf03r7k1", + "score": 0.39035138596571384, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 38, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 38, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 42, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uau01yq8kc9ara0a13p", + "score": 0.39050281163664174, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: i" + }, + { + "i": 5, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c4-r2.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c4-r2.json new file mode 100644 index 0000000..f5a7704 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c4-r2.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "cluster12c4-r2", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "20/20", + "goldTop10": "20/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 73, + "deepLatencyMs": 47, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 52, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33814625000000004, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 42, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jag00518kc97v7aiok2", + "score": 0.3899548569950754, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 47, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4jws00938kc9l5d2ca5p", + "score": 0.39057304910160945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 51, + "deepLatencyMs": 47, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 47, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 47, + "deepLatencyMs": 53, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 41, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mjz00p88kc9jgz2p1n4", + "score": 0.39016156237845945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 48, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 46, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4p4s01468kc93dp1x5it", + "score": 0.38992495978753283, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3413975, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 49, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.33770472222222225, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 47, + "deepLatencyMs": 51, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4rpe01jn8kc9hf03r7k1", + "score": 0.39035138596571384, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 47, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": 4, + "rankInPool": 4, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uau01yq8kc9ara0a13p", + "score": 0.39050281163664174, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: i" + }, + { + "i": 5, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c8-r1.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c8-r1.json new file mode 100644 index 0000000..20f5ba7 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c8-r1.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "cluster12c8-r1", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "20/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 73, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 57, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33814625000000004, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 53, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jag00518kc97v7aiok2", + "score": 0.3899548569950754, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 46, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4jws00938kc9l5d2ca5p", + "score": 0.39057304910160945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 44, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4mjz00p88kc9jgz2p1n4", + "score": 0.39016156237845945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 47, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4p4s01468kc93dp1x5it", + "score": 0.38992495978753283, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3413975, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 39, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.33770472222222225, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 38, + "deepLatencyMs": 42, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4rpe01jn8kc9hf03r7k1", + "score": 0.39035138596571384, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 47, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 45, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uau01yq8kc9ara0a13p", + "score": 0.39050281163664174, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: i" + }, + { + "i": 9, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c8-r2.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c8-r2.json new file mode 100644 index 0000000..a9aff9d --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster12c8-r2.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "cluster12c8-r2", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "20/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 80, + "deepLatencyMs": 57, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 47, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33814625000000004, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 42, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jag00518kc97v7aiok2", + "score": 0.3899548569950754, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4jws00938kc9l5d2ca5p", + "score": 0.39057304910160945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 46, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 47, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 44, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4mjz00p88kc9jgz2p1n4", + "score": 0.39016156237845945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 43, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4p4s01468kc93dp1x5it", + "score": 0.38992495978753283, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3413975, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 48, + "deepLatencyMs": 48, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.33770472222222225, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 43, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4rpe01jn8kc9hf03r7k1", + "score": 0.39035138596571384, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 46, + "deepLatencyMs": 47, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 49, + "deepLatencyMs": 49, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": 8, + "rankInPool": 8, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 46, + "deepLatencyMs": 46, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uau01yq8kc9ara0a13p", + "score": 0.39050281163664174, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: i" + }, + { + "i": 9, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster20-r1.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster20-r1.json new file mode 100644 index 0000000..983bf49 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster20-r1.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "cluster20-r1", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "20/20", + "goldTop10": "20/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 77, + "deepLatencyMs": 56, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 44, + "deepLatencyMs": 37, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33814625000000004, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3380996370967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 55, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jag00518kc97v7aiok2", + "score": 0.3899548569950754, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 43, + "deepLatencyMs": 48, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4jws00938kc9l5d2ca5p", + "score": 0.39057304910160945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 43, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 44, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.33670624423963136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 40, + "deepLatencyMs": 40, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mjz00p88kc9jgz2p1n4", + "score": 0.39016156237845945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.33670624423963136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 38, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 47, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.34092243150684937, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 43, + "deepLatencyMs": 45, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4p4s01468kc93dp1x5it", + "score": 0.38992495978753283, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3413975, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3413508870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 47, + "deepLatencyMs": 47, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 44, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 44, + "deepLatencyMs": 50, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.33770472222222225, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3376711815068494, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 52, + "deepLatencyMs": 51, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rpe01jn8kc9hf03r7k1", + "score": 0.39035138596571384, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 46, + "deepLatencyMs": 49, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 46, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 38, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4uau01yq8kc9ara0a13p", + "score": 0.39050281163664174, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: i" + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.34092243150684937, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.24053470472889169, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster20-r2.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster20-r2.json new file mode 100644 index 0000000..620ddcd --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated-backfill/cluster20-r2.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "cluster20-r2", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "20/20", + "goldTop10": "20/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 68, + "deepLatencyMs": 50, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 52, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33814625000000004, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3380996370967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 40, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jag00518kc97v7aiok2", + "score": 0.3899548569950754, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 40, + "deepLatencyMs": 44, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4jws00938kc9l5d2ca5p", + "score": 0.39057304910160945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 44, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 43, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 45, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.33670624423963136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 39, + "deepLatencyMs": 40, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mjz00p88kc9jgz2p1n4", + "score": 0.39016156237845945, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3367528571428571, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.33670624423963136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 49, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 45, + "deepLatencyMs": 49, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 46, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.34092243150684937, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 46, + "deepLatencyMs": 45, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4p4s01468kc93dp1x5it", + "score": 0.38992495978753283, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.3413975, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3413508870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 43, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.33770472222222225, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3376711815068494, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 44, + "deepLatencyMs": 42, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rpe01jn8kc9hf03r7k1", + "score": 0.39035138596571384, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 36, + "deepLatencyMs": 38, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 44, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 44, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.33489500000000005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.3348483870967742, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": 2, + "rankInPool": 2, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 45, + "deepLatencyMs": 42, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4uau01yq8kc9ara0a13p", + "score": 0.39050281163664174, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: i" + }, + { + "i": 3, + "id": "cmt6a4u8o01yb8kc98lqh2rkz", + "score": 0.34095597222222224, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.34092243150684937, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.24053470472889169, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth10-r1.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth10-r1.json new file mode 100644 index 0000000..8a531da --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth10-r1.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "depth10-r1", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "14/20", + "goldAnywhereInPool": "14/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 73, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4ij700098kc9vkuejjt5", + "score": 0.3906301178996782, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 48, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j7m004c8kc9gzgak8rf", + "score": 0.39002603494640714, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 46, + "deepLatencyMs": 40, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jt1008d8kc94097tif0", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jto008j8kc97a8czmex", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 38, + "deepLatencyMs": 40, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kf500cc8kc9nqyij4wr", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4khg00cm8kc9fbhjpmhe", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 43, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l3r00fn8kc97ksn5zix", + "score": 0.3905300025675409, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 42, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lsd00kc8kc96ozo34ls", + "score": 0.39012167988138485, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 39, + "deepLatencyMs": 39, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mgj00od8kc9kvoukeul", + "score": 0.39016066283438616, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 35, + "deepLatencyMs": 36, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4n1r00se8kc9rpfauix3", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4n3u00ss8kc9dib989cz", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 38, + "deepLatencyMs": 32, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nog00vr8kc9ffabvuw1", + "score": 0.3902888026389062, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 30, + "deepLatencyMs": 34, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4ocw00zo8kc9rn1tf6l3", + "score": 0.3889848715893484, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 34, + "deepLatencyMs": 39, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4ozv013d8kc9r8iyyigl", + "score": 0.3903794721732559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 36, + "deepLatencyMs": 34, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pns01748kc943hu0yb7", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pq2017j8kc9ybviz9x0", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 38, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4qal01b08kc9zaauphup", + "score": 0.3903116175839953, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 37, + "deepLatencyMs": 37, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qx601f28kc9fppghxkl", + "score": 0.38871115857215094, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 40, + "deepLatencyMs": 36, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rlo01iw8kc9659pegeb", + "score": 0.38873973474135837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 39, + "deepLatencyMs": 43, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s9k01mp8kc915oipcnw", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4sbv01n48kc9xjtg7rlm", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 37, + "deepLatencyMs": 37, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sxa01qa8kc913es7ahw", + "score": 0.3902010068623859, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 40, + "deepLatencyMs": 39, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tju01u88kc9tsh17i6r", + "score": 0.3896808146024071, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 36, + "deepLatencyMs": 36, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3899757441396751, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 39, + "deepLatencyMs": 37, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth10-r2.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth10-r2.json new file mode 100644 index 0000000..8498c53 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth10-r2.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "depth10-r2", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "14/20", + "goldAnywhereInPool": "14/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 69, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4ij700098kc9vkuejjt5", + "score": 0.3906301178996782, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 47, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j7m004c8kc9gzgak8rf", + "score": 0.39002603494640714, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jt1008d8kc94097tif0", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jto008j8kc97a8czmex", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 41, + "deepLatencyMs": 44, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kf500cc8kc9nqyij4wr", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4khg00cm8kc9fbhjpmhe", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 43, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l3r00fn8kc97ksn5zix", + "score": 0.3905300025675409, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 41, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lsd00kc8kc96ozo34ls", + "score": 0.39012167988138485, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 42, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mgj00od8kc9kvoukeul", + "score": 0.39016066283438616, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 40, + "deepLatencyMs": 41, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4n1r00se8kc9rpfauix3", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4n3u00ss8kc9dib989cz", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 41, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nog00vr8kc9ffabvuw1", + "score": 0.3902888026389062, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 40, + "deepLatencyMs": 39, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4ocw00zo8kc9rn1tf6l3", + "score": 0.3889848715893484, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 43, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4ozv013d8kc9r8iyyigl", + "score": 0.3903794721732559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 45, + "deepLatencyMs": 42, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pns01748kc943hu0yb7", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pq2017j8kc9ybviz9x0", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 43, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4qal01b08kc9zaauphup", + "score": 0.3903116175839953, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 43, + "deepLatencyMs": 36, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qx601f28kc9fppghxkl", + "score": 0.38871115857215094, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 43, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rlo01iw8kc9659pegeb", + "score": 0.38873973474135837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 48, + "deepLatencyMs": 43, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s9k01mp8kc915oipcnw", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4sbv01n48kc9xjtg7rlm", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 39, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sxa01qa8kc913es7ahw", + "score": 0.3902010068623859, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 44, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tju01u88kc9tsh17i6r", + "score": 0.3896808146024071, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 40, + "deepLatencyMs": 38, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3899757441396751, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": -1, + "rankInPool": -1, + "pageSize": 10, + "poolSize": 10, + "pageLatencyMs": 40, + "deepLatencyMs": 42, + "goldScore": null, + "goldBand": "ABSENT", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth12-r1.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth12-r1.json new file mode 100644 index 0000000..f9c65a9 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth12-r1.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "depth12-r1", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "14/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 69, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4ij700098kc9vkuejjt5", + "score": 0.3906301178996782, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 72, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j7m004c8kc9gzgak8rf", + "score": 0.39002603494640714, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 42, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jt1008d8kc94097tif0", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jto008j8kc97a8czmex", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 41, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kf500cc8kc9nqyij4wr", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4khg00cm8kc9fbhjpmhe", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l3r00fn8kc97ksn5zix", + "score": 0.3905300025675409, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lsd00kc8kc96ozo34ls", + "score": 0.39012167988138485, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mgj00od8kc9kvoukeul", + "score": 0.39016066283438616, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 44, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4n1r00se8kc9rpfauix3", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4n3u00ss8kc9dib989cz", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 47, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nog00vr8kc9ffabvuw1", + "score": 0.3902888026389062, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4ocw00zo8kc9rn1tf6l3", + "score": 0.3889848715893484, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 48, + "deepLatencyMs": 47, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4ozv013d8kc9r8iyyigl", + "score": 0.3903794721732559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 44, + "deepLatencyMs": 42, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pns01748kc943hu0yb7", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pq2017j8kc9ybviz9x0", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4qal01b08kc9zaauphup", + "score": 0.3903116175839953, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qx601f28kc9fppghxkl", + "score": 0.38871115857215094, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rlo01iw8kc9659pegeb", + "score": 0.38873973474135837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 41, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s9k01mp8kc915oipcnw", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4sbv01n48kc9xjtg7rlm", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sxa01qa8kc913es7ahw", + "score": 0.3902010068623859, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tju01u88kc9tsh17i6r", + "score": 0.3896808146024071, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3899757441396751, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 41, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth12-r2.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth12-r2.json new file mode 100644 index 0000000..c4a890f --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth12-r2.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "depth12-r2", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "14/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 70, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4ij700098kc9vkuejjt5", + "score": 0.3906301178996782, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 49, + "deepLatencyMs": 48, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j7m004c8kc9gzgak8rf", + "score": 0.39002603494640714, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 43, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jt1008d8kc94097tif0", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jto008j8kc97a8czmex", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 43, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kf500cc8kc9nqyij4wr", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4khg00cm8kc9fbhjpmhe", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l3r00fn8kc97ksn5zix", + "score": 0.3905300025675409, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lsd00kc8kc96ozo34ls", + "score": 0.39012167988138485, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mgj00od8kc9kvoukeul", + "score": 0.39016066283438616, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 38, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4n1r00se8kc9rpfauix3", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4n3u00ss8kc9dib989cz", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nog00vr8kc9ffabvuw1", + "score": 0.3902888026389062, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 37, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4ocw00zo8kc9rn1tf6l3", + "score": 0.3889848715893484, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4ozv013d8kc9r8iyyigl", + "score": 0.3903794721732559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 41, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pns01748kc943hu0yb7", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pq2017j8kc9ybviz9x0", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 41, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4qal01b08kc9zaauphup", + "score": 0.3903116175839953, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qx601f28kc9fppghxkl", + "score": 0.38871115857215094, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 40, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rlo01iw8kc9659pegeb", + "score": 0.38873973474135837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 46, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s9k01mp8kc915oipcnw", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4sbv01n48kc9xjtg7rlm", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sxa01qa8kc913es7ahw", + "score": 0.3902010068623859, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 43, + "deepLatencyMs": 49, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tju01u88kc9tsh17i6r", + "score": 0.3896808146024071, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 39, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3899757441396751, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 12, + "pageLatencyMs": 45, + "deepLatencyMs": 47, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth20-r1.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth20-r1.json new file mode 100644 index 0000000..e478250 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth20-r1.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "depth20-r1", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "14/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 69, + "deepLatencyMs": 50, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4ij700098kc9vkuejjt5", + "score": 0.3906301178996782, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 49, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j7m004c8kc9gzgak8rf", + "score": 0.39002603494640714, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 43, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jt1008d8kc94097tif0", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jto008j8kc97a8czmex", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 43, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kf500cc8kc9nqyij4wr", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4khg00cm8kc9fbhjpmhe", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 46, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l3r00fn8kc97ksn5zix", + "score": 0.3905300025675409, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 43, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lsd00kc8kc96ozo34ls", + "score": 0.39012167988138485, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mgj00od8kc9kvoukeul", + "score": 0.39016066283438616, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 37, + "deepLatencyMs": 39, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4n1r00se8kc9rpfauix3", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4n3u00ss8kc9dib989cz", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nog00vr8kc9ffabvuw1", + "score": 0.3902888026389062, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4ocw00zo8kc9rn1tf6l3", + "score": 0.3889848715893484, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 43, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4ozv013d8kc9r8iyyigl", + "score": 0.3903794721732559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 40, + "deepLatencyMs": 40, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pns01748kc943hu0yb7", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pq2017j8kc9ybviz9x0", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 39, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4qal01b08kc9zaauphup", + "score": 0.3903116175839953, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 37, + "deepLatencyMs": 30, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qx601f28kc9fppghxkl", + "score": 0.38871115857215094, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 37, + "deepLatencyMs": 37, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rlo01iw8kc9659pegeb", + "score": 0.38873973474135837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 38, + "deepLatencyMs": 41, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s9k01mp8kc915oipcnw", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4sbv01n48kc9xjtg7rlm", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 37, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sxa01qa8kc913es7ahw", + "score": 0.3902010068623859, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 40, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tju01u88kc9tsh17i6r", + "score": 0.3896808146024071, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 29, + "deepLatencyMs": 39, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3899757441396751, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 43, + "deepLatencyMs": 44, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth20-r2.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth20-r2.json new file mode 100644 index 0000000..c7b11b9 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth20-r2.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "depth20-r2", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "14/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 69, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4ij700098kc9vkuejjt5", + "score": 0.3906301178996782, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 49, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j7m004c8kc9gzgak8rf", + "score": 0.39002603494640714, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 43, + "deepLatencyMs": 40, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jt1008d8kc94097tif0", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jto008j8kc97a8czmex", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 42, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kf500cc8kc9nqyij4wr", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4khg00cm8kc9fbhjpmhe", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l3r00fn8kc97ksn5zix", + "score": 0.3905300025675409, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lsd00kc8kc96ozo34ls", + "score": 0.39012167988138485, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mgj00od8kc9kvoukeul", + "score": 0.39016066283438616, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 40, + "deepLatencyMs": 42, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4n1r00se8kc9rpfauix3", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4n3u00ss8kc9dib989cz", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nog00vr8kc9ffabvuw1", + "score": 0.3902888026389062, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 40, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4ocw00zo8kc9rn1tf6l3", + "score": 0.3889848715893484, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 41, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4ozv013d8kc9r8iyyigl", + "score": 0.3903794721732559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 39, + "deepLatencyMs": 44, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pns01748kc943hu0yb7", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pq2017j8kc9ybviz9x0", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 39, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4qal01b08kc9zaauphup", + "score": 0.3903116175839953, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 37, + "deepLatencyMs": 39, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qx601f28kc9fppghxkl", + "score": 0.38871115857215094, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 38, + "deepLatencyMs": 50, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rlo01iw8kc9659pegeb", + "score": 0.38873973474135837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 38, + "deepLatencyMs": 40, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s9k01mp8kc915oipcnw", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4sbv01n48kc9xjtg7rlm", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 40, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sxa01qa8kc913es7ahw", + "score": 0.3902010068623859, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 40, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tju01u88kc9tsh17i6r", + "score": 0.3896808146024071, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 37, + "deepLatencyMs": 38, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3899757441396751, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 20, + "pageLatencyMs": 42, + "deepLatencyMs": 43, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth50-r1.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth50-r1.json new file mode 100644 index 0000000..4b8fd75 --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth50-r1.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "depth50-r1", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "14/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 66, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4ij700098kc9vkuejjt5", + "score": 0.3906301178996782, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 49, + "deepLatencyMs": 48, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j7m004c8kc9gzgak8rf", + "score": 0.39002603494640714, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 40, + "deepLatencyMs": 41, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jt1008d8kc94097tif0", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jto008j8kc97a8czmex", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 43, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kf500cc8kc9nqyij4wr", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4khg00cm8kc9fbhjpmhe", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l3r00fn8kc97ksn5zix", + "score": 0.3905300025675409, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 45, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lsd00kc8kc96ozo34ls", + "score": 0.39012167988138485, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 43, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mgj00od8kc9kvoukeul", + "score": 0.39016066283438616, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 39, + "deepLatencyMs": 41, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4n1r00se8kc9rpfauix3", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4n3u00ss8kc9dib989cz", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nog00vr8kc9ffabvuw1", + "score": 0.3902888026389062, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 40, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4ocw00zo8kc9rn1tf6l3", + "score": 0.3889848715893484, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 43, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4ozv013d8kc9r8iyyigl", + "score": 0.3903794721732559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 40, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pns01748kc943hu0yb7", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pq2017j8kc9ybviz9x0", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4qal01b08kc9zaauphup", + "score": 0.3903116175839953, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 40, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qx601f28kc9fppghxkl", + "score": 0.38871115857215094, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rlo01iw8kc9659pegeb", + "score": 0.38873973474135837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 40, + "deepLatencyMs": 44, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s9k01mp8kc915oipcnw", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4sbv01n48kc9xjtg7rlm", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 38, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sxa01qa8kc913es7ahw", + "score": 0.3902010068623859, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tju01u88kc9tsh17i6r", + "score": 0.3896808146024071, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 39, + "deepLatencyMs": 40, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3899757441396751, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth50-r2.json b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth50-r2.json new file mode 100644 index 0000000..a51496c --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated/depth50-r2.json @@ -0,0 +1,1915 @@ +{ + "summary": { + "label": "depth50-r2", + "baseUrl": "http://127.0.0.1:3017", + "limit": 10, + "headers": {}, + "tasks": 20, + "goldTop1": "14/20", + "goldTop5": "14/20", + "goldTop10": "14/20", + "goldAnywhereInPool": "20/20" + }, + "rows": [ + { + "task": "create-user-endpoint", + "query": "MNEMON benchmark project fact for user creation API endpoint schema and validation rules", + "goldId": "cmt6a4ifu00008kc9vqa8ho5y", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 67, + "deepLatencyMs": 44, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4ifu00008kc9vqa8ho5y", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 1, + "id": "cmt6a4igr00018kc99i5i66qg", + "score": 0.3911175556876966, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 2, + "id": "cmt6a4ih000028kc9zgolat1x", + "score": 0.39104985599491626, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 3, + "id": "cmt6a4ih800038kc92mlwuyz0", + "score": 0.3909842719175353, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 4, + "id": "cmt6a4ihe00048kc9mk3ybqz7", + "score": 0.3909207058117661, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 5, + "id": "cmt6a4ihm00058kc98q0cfifn", + "score": 0.39085906595162634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 6, + "id": "cmt6a4ii500068kc94p54l6us", + "score": 0.3907992660873115, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 7, + "id": "cmt6a4iij00078kc97oobusxn", + "score": 0.3907412250425354, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 8, + "id": "cmt6a4iiu00088kc99ll6eucq", + "score": 0.39068486634688315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + }, + { + "i": 9, + "id": "cmt6a4ij700098kc9vkuejjt5", + "score": 0.3906301178996782, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for user creation API endpoint schema an" + } + ] + }, + { + "task": "order-refund-handler", + "query": "MNEMON benchmark project fact for order refund policy and payment integration", + "goldId": "cmt6a4iqw001i8kc90boz30m3", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 49, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4iqw001i8kc90boz30m3", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 1, + "id": "cmt6a4is9001q8kc9a2pi74zh", + "score": 0.39051347273442555, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 2, + "id": "cmt6a4its001v8kc9rgqblp9z", + "score": 0.39044577304164524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 3, + "id": "cmt6a4ivd00268kc920p3n8by", + "score": 0.3903801889642643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 4, + "id": "cmt6a4ix2002g8kc9u9l725hn", + "score": 0.390316622858495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 5, + "id": "cmt6a4izq002y8kc9kp14k0kd", + "score": 0.39025498299835526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 6, + "id": "cmt6a4j1n00378kc92cb8o1bt", + "score": 0.39019518313404045, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 7, + "id": "cmt6a4j42003g8kc9l6ssvs7e", + "score": 0.39013714208926425, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 8, + "id": "cmt6a4j5s003w8kc9x6vfi012", + "score": 0.3900807833936121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + }, + { + "i": 9, + "id": "cmt6a4j7m004c8kc9gzgak8rf", + "score": 0.39002603494640714, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order refund policy and payment inte" + } + ] + }, + { + "task": "paginated-list-endpoint", + "query": "MNEMON benchmark project fact for pagination conventions for list endpoints", + "goldId": "cmt6a4jag00518kc97v7aiok2", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 41, + "deepLatencyMs": 42, + "goldScore": 0.3899548569950754, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jda005f8kc9nfxhqm6w", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 1, + "id": "cmt6a4jff005q8kc97t0j8vnu", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 2, + "id": "cmt6a4jhm00688kc9xw9ptguj", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 3, + "id": "cmt6a4jkc006l8kc9ponujkl5", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 4, + "id": "cmt6a4jls006x8kc91oadpvrl", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 5, + "id": "cmt6a4jnb00758kc96gy9bm5e", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 6, + "id": "cmt6a4jp6007g8kc90gyasol5", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 7, + "id": "cmt6a4jr0007x8kc93wu05ca0", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 8, + "id": "cmt6a4jt1008d8kc94097tif0", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + }, + { + "i": 9, + "id": "cmt6a4jto008j8kc97a8czmex", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for pagination conventions for list endp" + } + ] + }, + { + "task": "webhook-processor", + "query": "MNEMON benchmark project fact for webhook signature verification and idempotency", + "goldId": "cmt6a4jws00938kc9l5d2ca5p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 44, + "deepLatencyMs": 46, + "goldScore": 0.39057304910160945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4jys009g8kc9wxqg1fko", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 1, + "id": "cmt6a4k13009t8kc900ke6ho7", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 2, + "id": "cmt6a4k2w00a58kc9z1g4vyfw", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 3, + "id": "cmt6a4k4t00ag8kc9t77s2i0v", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 4, + "id": "cmt6a4k7300ap8kc9j8s6a7ph", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 5, + "id": "cmt6a4k8u00b08kc96fofub7c", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 6, + "id": "cmt6a4kb200bh8kc9hwuctcam", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 7, + "id": "cmt6a4kdf00bx8kc9l7lzqexm", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 8, + "id": "cmt6a4kf500cc8kc9nqyij4wr", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + }, + { + "i": 9, + "id": "cmt6a4khg00cm8kc9fbhjpmhe", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for webhook signature verification and i" + } + ] + }, + { + "task": "rate-limiter-middleware", + "query": "MNEMON benchmark project fact for rate limiting middleware and Redis keys", + "goldId": "cmt6a4kjv00cv8kc98db9crcq", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 49, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4kjv00cv8kc98db9crcq", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 1, + "id": "cmt6a4kmb00d98kc9rpyel6ek", + "score": 0.39101744035555924, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 2, + "id": "cmt6a4koz00di8kc929xjhng2", + "score": 0.39094974066277893, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 3, + "id": "cmt6a4krl00dw8kc9yr4lzubd", + "score": 0.390884156585398, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 4, + "id": "cmt6a4ktt00e98kc94vzdbj5l", + "score": 0.3908205904796287, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 5, + "id": "cmt6a4kvm00eh8kc98krt5d9u", + "score": 0.39075895061948884, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 6, + "id": "cmt6a4kxm00et8kc9loh5udgt", + "score": 0.39069915075517414, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 7, + "id": "cmt6a4kzl00f48kc9s21r5xgw", + "score": 0.39064110971039795, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 8, + "id": "cmt6a4l1q00fe8kc9osu7dtxd", + "score": 0.3905847510147458, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + }, + { + "i": 9, + "id": "cmt6a4l3r00fn8kc97ksn5zix", + "score": 0.3905300025675409, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for rate limiting middleware and Redis k" + } + ] + }, + { + "task": "session-token-rotation", + "query": "MNEMON benchmark project fact for session token rotation and the session store", + "goldId": "cmt6a4l8k00gq8kc927matnui", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 42, + "deepLatencyMs": 46, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4l8k00gq8kc927matnui", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 1, + "id": "cmt6a4lb700h88kc9aib7w2zi", + "score": 0.3906091176694033, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 2, + "id": "cmt6a4ldt00hp8kc9f8qo4wuy", + "score": 0.39054141797662295, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 3, + "id": "cmt6a4lg200hy8kc9m8rvphyd", + "score": 0.390475833899242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 4, + "id": "cmt6a4lhw00i68kc9ddlcehh8", + "score": 0.3904122677934728, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 5, + "id": "cmt6a4ljw00in8kc95cdh899r", + "score": 0.39035062793333286, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 6, + "id": "cmt6a4lm200j38kc94oeavgoe", + "score": 0.39029082806901816, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 7, + "id": "cmt6a4lo300ji8kc9ljkg7ohc", + "score": 0.390232787024242, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 8, + "id": "cmt6a4lqf00jw8kc93ayv7ozj", + "score": 0.39017642832858984, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + }, + { + "i": 9, + "id": "cmt6a4lsd00kc8kc96ozo34ls", + "score": 0.39012167988138485, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for session token rotation and the sessi" + } + ] + }, + { + "task": "session-cache-ttl", + "query": "MNEMON benchmark project fact for the session cache TTL and cache keys", + "goldId": "cmt6a4lwj00l68kc9630ig6ps", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 43, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4lwj00l68kc9630ig6ps", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 1, + "id": "cmt6a4lz800li8kc9dwlpbbdd", + "score": 0.39064810062240457, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 2, + "id": "cmt6a4m2400lv8kc9ixf4c7py", + "score": 0.3905804009296243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 3, + "id": "cmt6a4m4s00m58kc94bex455y", + "score": 0.3905148168522434, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 4, + "id": "cmt6a4m6r00mi8kc920yq42k2", + "score": 0.39045125074647413, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 5, + "id": "cmt6a4m8o00mu8kc9qkx5691b", + "score": 0.3903896108863343, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 6, + "id": "cmt6a4mar00nb8kc9u2chds0s", + "score": 0.39032981102201947, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 7, + "id": "cmt6a4mcv00nm8kc918ur3qzh", + "score": 0.39027176997724333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 8, + "id": "cmt6a4men00o38kc9hyhhuvvl", + "score": 0.3902154112815912, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + }, + { + "i": 9, + "id": "cmt6a4mgj00od8kc9kvoukeul", + "score": 0.39016066283438616, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for the session cache TTL and cache keys" + } + ] + }, + { + "task": "inventory-reservation", + "query": "MNEMON benchmark project fact for inventory reservation and the inventory client", + "goldId": "cmt6a4mjz00p88kc9jgz2p1n4", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 41, + "deepLatencyMs": 41, + "goldScore": 0.39016156237845945, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4mm000pm8kc92fj1hs5z", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 1, + "id": "cmt6a4mnx00ps8kc9bdc3pwo1", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 2, + "id": "cmt6a4mov00pz8kc9dh4a38uv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 3, + "id": "cmt6a4mrr00qf8kc9qx7fgh5j", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 4, + "id": "cmt6a4mtv00qw8kc905vn58e0", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 5, + "id": "cmt6a4mvs00rc8kc95bmgsh8s", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 6, + "id": "cmt6a4mxo00ro8kc9znl9k5gm", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 7, + "id": "cmt6a4mzt00s38kc9kwhd0e5d", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 8, + "id": "cmt6a4n1r00se8kc9rpfauix3", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + }, + { + "i": 9, + "id": "cmt6a4n3u00ss8kc9dib989cz", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for inventory reservation and the invent" + } + ] + }, + { + "task": "welcome-email-dispatch", + "query": "MNEMON benchmark project fact for welcome email templates and mailer dispatch", + "goldId": "cmt6a4n6600t08kc9rw3glzh1", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 33, + "deepLatencyMs": 43, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4n6600t08kc9rw3glzh1", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 1, + "id": "cmt6a4n8700ta8kc90kfr5ot1", + "score": 0.3907762404269247, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 2, + "id": "cmt6a4naf00tn8kc92rsa2hx2", + "score": 0.39070854073414435, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 3, + "id": "cmt6a4ncr00tz8kc9pf9a4aq4", + "score": 0.39064295665676335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 4, + "id": "cmt6a4nex00ua8kc9p3mck7g7", + "score": 0.39057939055099417, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 5, + "id": "cmt6a4nh100uk8kc92xnuwjck", + "score": 0.39051775069085426, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 6, + "id": "cmt6a4nj900uy8kc953g9fita", + "score": 0.39045795082653945, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 7, + "id": "cmt6a4nk900va8kc9q3gappr9", + "score": 0.3903999097817634, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 8, + "id": "cmt6a4nmo00vj8kc9dj4snykt", + "score": 0.39034355108611124, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + }, + { + "i": 9, + "id": "cmt6a4nog00vr8kc9ffabvuw1", + "score": 0.3902888026389062, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for welcome email templates and mailer d" + } + ] + }, + { + "task": "beta-checkout-flag", + "query": "MNEMON benchmark project fact for feature flag keys and evaluation", + "goldId": "cmt6a4nsl00wm8kc9n5vpspnt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 28, + "deepLatencyMs": 32, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4nsl00wm8kc9n5vpspnt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: fl" + }, + { + "i": 1, + "id": "cmt6a4nun00x08kc9qai7ljeo", + "score": 0.3894723093773669, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 2, + "id": "cmt6a4nx500xa8kc945wf7hyl", + "score": 0.3894046096845865, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 3, + "id": "cmt6a4ny400xj8kc9jzkl5qna", + "score": 0.38933902560720557, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 4, + "id": "cmt6a4o0z00xs8kc9jli2ifys", + "score": 0.3892754595014364, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 5, + "id": "cmt6a4o3n00y98kc9buruqc3u", + "score": 0.3892138196412964, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 6, + "id": "cmt6a4o5t00yp8kc9w3pou4vh", + "score": 0.3891540197769817, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 7, + "id": "cmt6a4o7v00z18kc9f8zq8p2f", + "score": 0.3890959787322056, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 8, + "id": "cmt6a4oae00z98kc9d0ysxkkt", + "score": 0.38903962003655335, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + }, + { + "i": 9, + "id": "cmt6a4ocw00zo8kc9rn1tf6l3", + "score": 0.3889848715893484, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for feature flag keys and evaluation: ar" + } + ] + }, + { + "task": "audit-event-logging", + "query": "MNEMON benchmark project fact for audit event logging and timestamps", + "goldId": "cmt6a4oh2010d8kc9yuqgzna0", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 32, + "deepLatencyMs": 42, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4oh2010d8kc9yuqgzna0", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 1, + "id": "cmt6a4oj9010q8kc9sy3436kx", + "score": 0.3908669099612743, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 2, + "id": "cmt6a4oli01108kc978sjx0aj", + "score": 0.39079921026849396, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 3, + "id": "cmt6a4ons01198kc912nr0z28", + "score": 0.390733626191113, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 4, + "id": "cmt6a4oq5011q8kc9hc7h43hn", + "score": 0.39067006008534383, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 5, + "id": "cmt6a4os701228kc9so3aa2hm", + "score": 0.39060842022520387, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 6, + "id": "cmt6a4oua012e8kc9av5xfeu3", + "score": 0.39054862036088916, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 7, + "id": "cmt6a4ovz012p8kc9citpt7lg", + "score": 0.3904905793161131, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 8, + "id": "cmt6a4oxr01358kc90vpm1fot", + "score": 0.39043422062046085, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + }, + { + "i": 9, + "id": "cmt6a4ozv013d8kc9r8iyyigl", + "score": 0.3903794721732559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for audit event logging and timestamps: " + } + ] + }, + { + "task": "currency-conversion", + "query": "MNEMON benchmark project fact for currency conversion and money units", + "goldId": "cmt6a4p4s01468kc93dp1x5it", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 31, + "deepLatencyMs": 32, + "goldScore": 0.38992495978753283, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4p77014j8kc9p908y6bw", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 1, + "id": "cmt6a4p99014v8kc94jkfm9ug", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 2, + "id": "cmt6a4pb801558kc9f8u9drwv", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 3, + "id": "cmt6a4pdf015e8kc95vxobnb0", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 4, + "id": "cmt6a4pfd015p8kc9n18vdebi", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 5, + "id": "cmt6a4pha015x8kc97dp6bht1", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 6, + "id": "cmt6a4pjr016e8kc972r4dsdd", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 7, + "id": "cmt6a4plo016u8kc9mqiefwx5", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 8, + "id": "cmt6a4pns01748kc943hu0yb7", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + }, + { + "i": 9, + "id": "cmt6a4pq2017j8kc9ybviz9x0", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for currency conversion and money units:" + } + ] + }, + { + "task": "report-object-upload", + "query": "MNEMON benchmark project fact for report uploads and object storage keys", + "goldId": "cmt6a4prv017s8kc9nuqjqz7p", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 32, + "deepLatencyMs": 34, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4prv017s8kc9nuqjqz7p", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 1, + "id": "cmt6a4pu501868kc92sprr4us", + "score": 0.3907990553720136, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 2, + "id": "cmt6a4pwa018j8kc90rna5t84", + "score": 0.3907313556792333, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 3, + "id": "cmt6a4pyg018v8kc9lxsii3k8", + "score": 0.39066577160185234, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 4, + "id": "cmt6a4q0m01968kc9zhiuyflh", + "score": 0.39060220549608315, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 5, + "id": "cmt6a4q2n019g8kc9c3nd5xdv", + "score": 0.39054056563594325, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 6, + "id": "cmt6a4q4u019u8kc97y1zeyt4", + "score": 0.3904807657716285, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 7, + "id": "cmt6a4q5y01ab8kc9cegkd5px", + "score": 0.3904227247268524, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 8, + "id": "cmt6a4q8g01ak8kc9zrw1uq5s", + "score": 0.39036636603120023, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + }, + { + "i": 9, + "id": "cmt6a4qal01b08kc9zaauphup", + "score": 0.3903116175839953, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for report uploads and object storage ke" + } + ] + }, + { + "task": "user-export-job", + "query": "MNEMON benchmark project fact for export jobs and queue naming", + "goldId": "cmt6a4qek01bn8kc9t5eq96rt", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 41, + "deepLatencyMs": 32, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4qek01bn8kc9t5eq96rt", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for export jobs and queue naming: queued" + }, + { + "i": 1, + "id": "cmt6a4qgf01c48kc93n81kil6", + "score": 0.3891985963601694, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 2, + "id": "cmt6a4qii01cc8kc9b1v63xh6", + "score": 0.3891308966673891, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 3, + "id": "cmt6a4qkg01cq8kc9eyz8e9ko", + "score": 0.3890653125900081, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 4, + "id": "cmt6a4qm101d08kc9ffos26u0", + "score": 0.38900174648423885, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 5, + "id": "cmt6a4qo201df8kc9tmcivrgh", + "score": 0.38894010662409895, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 6, + "id": "cmt6a4qpd01dq8kc97pccpqza", + "score": 0.3888803067597842, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 7, + "id": "cmt6a4qsc01e88kc9w6h9v4ut", + "score": 0.38882226571500805, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 8, + "id": "cmt6a4qum01en8kc9y60xdxwt", + "score": 0.3887659070193559, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + }, + { + "i": 9, + "id": "cmt6a4qx601f28kc9fppghxkl", + "score": 0.38871115857215094, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for export jobs and queue naming: archiv" + } + ] + }, + { + "task": "http-metrics-recording", + "query": "MNEMON benchmark project fact for HTTP metrics names and labels", + "goldId": "cmt6a4r1f01fs8kc9vl4jfh34", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 39, + "deepLatencyMs": 41, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4r1f01fs8kc9vl4jfh34", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: use m" + }, + { + "i": 1, + "id": "cmt6a4r3l01g68kc9hrxp8a70", + "score": 0.3892271725293768, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 2, + "id": "cmt6a4r5m01gj8kc9mp1zet2u", + "score": 0.38915947283659647, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 3, + "id": "cmt6a4r8301gs8kc9dxpf3pif", + "score": 0.38909388875921547, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 4, + "id": "cmt6a4ral01h98kc9pd96etpr", + "score": 0.3890303226534463, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 5, + "id": "cmt6a4rd701hp8kc9d4u093d9", + "score": 0.38896868279330643, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 6, + "id": "cmt6a4rfc01i18kc9pi81uiuq", + "score": 0.38890888292899173, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 7, + "id": "cmt6a4rhe01ig8kc9lg89fv19", + "score": 0.3888508418842156, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 8, + "id": "cmt6a4rja01ir8kc9n6sg4kf4", + "score": 0.3887944831885633, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + }, + { + "i": 9, + "id": "cmt6a4rlo01iw8kc9659pegeb", + "score": 0.38873973474135837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP metrics names and labels: archi" + } + ] + }, + { + "task": "tenant-scoped-projects", + "query": "MNEMON benchmark project fact for tenant scoping and TenantId handling", + "goldId": "cmt6a4rpe01jn8kc9hf03r7k1", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 32, + "deepLatencyMs": 41, + "goldScore": 0.39035138596571384, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4rrv01k18kc90st7s18f", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 1, + "id": "cmt6a4rtx01ke8kc9m5jo5kqv", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 2, + "id": "cmt6a4rwb01kq8kc9uqg1772w", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 3, + "id": "cmt6a4rys01l18kc9l04qh9ye", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 4, + "id": "cmt6a4s0w01la8kc9nppaweia", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 5, + "id": "cmt6a4s3f01lj8kc90dtbvule", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 6, + "id": "cmt6a4s5i01m18kc9sjsa0eax", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 7, + "id": "cmt6a4s7h01m98kc9m56acdxs", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 8, + "id": "cmt6a4s9k01mp8kc915oipcnw", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + }, + { + "i": 9, + "id": "cmt6a4sbv01n48kc9xjtg7rlm", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for tenant scoping and TenantId handling" + } + ] + }, + { + "task": "alias-domain-errors", + "query": "MNEMON benchmark project fact for alias domain errors and error codes", + "goldId": "cmt6a4sdz01nd8kc9s77mfmke", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 32, + "deepLatencyMs": 30, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4sdz01nd8kc9s77mfmke", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 1, + "id": "cmt6a4sgd01nl8kc9v0o3vuxw", + "score": 0.39068844465040437, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 2, + "id": "cmt6a4siw01nz8kc9i9nv9mwp", + "score": 0.390620744957624, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 3, + "id": "cmt6a4sl801oc8kc9x8vgrjnq", + "score": 0.390555160880243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 4, + "id": "cmt6a4snd01oo8kc97gaeth7f", + "score": 0.3904915947744738, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 5, + "id": "cmt6a4spr01oz8kc96kdotl8e", + "score": 0.39042995491433397, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 6, + "id": "cmt6a4srr01p98kc9pxef9hga", + "score": 0.39037015505001915, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 7, + "id": "cmt6a4stt01pi8kc9fut1jcg5", + "score": 0.390312114005243, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 8, + "id": "cmt6a4sw301pw8kc92o8p8xxx", + "score": 0.3902557553095909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + }, + { + "i": 9, + "id": "cmt6a4sxa01qa8kc913es7ahw", + "score": 0.3902010068623859, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for alias domain errors and error codes:" + } + ] + }, + { + "task": "app-config-parsing", + "query": "MNEMON benchmark project fact for app config environment variables and defaults", + "goldId": "cmt6a4t1u01r78kc9x7wq726e", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 45, + "deepLatencyMs": 31, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4t1u01r78kc9x7wq726e", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 1, + "id": "cmt6a4t3m01rm8kc9dg4yj9wi", + "score": 0.3901682523904256, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 2, + "id": "cmt6a4t5701s08kc9vbktx1g4", + "score": 0.39010055269764526, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 3, + "id": "cmt6a4t7701sd8kc9zbc9bwny", + "score": 0.3900349686202642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 4, + "id": "cmt6a4t9b01sp8kc99rzsggvp", + "score": 0.389971402514495, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 5, + "id": "cmt6a4tba01sx8kc994jysyge", + "score": 0.3899097626543551, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 6, + "id": "cmt6a4tde01t88kc96zbvl1l6", + "score": 0.38984996279004036, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 7, + "id": "cmt6a4tfq01tp8kc9keld9ddf", + "score": 0.3897919217452642, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 8, + "id": "cmt6a4thi01ty8kc9jmksusx6", + "score": 0.3897355630496121, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + }, + { + "i": 9, + "id": "cmt6a4tju01u88kc9tsh17i6r", + "score": 0.3896808146024071, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for app config environment variables and" + } + ] + }, + { + "task": "http-request-logging", + "query": "MNEMON benchmark project fact for HTTP request logging levels and events", + "goldId": "cmt6a4tnr01v38kc9wqh5jy0s", + "rankInPage": 0, + "rankInPool": 0, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 38, + "deepLatencyMs": 30, + "goldScore": 0.39125, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4tnr01v38kc9wqh5jy0s", + "score": 0.39125, + "band": "vector<=1.00", + "gold": true, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 1, + "id": "cmt6a4tps01vc8kc93myzaxo4", + "score": 0.3904631819276935, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 2, + "id": "cmt6a4trv01vq8kc9j2yx754d", + "score": 0.39039548223491316, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 3, + "id": "cmt6a4ttw01w38kc9mllywmkx", + "score": 0.3903298981575323, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 4, + "id": "cmt6a4twh01wf8kc9bo7pidd8", + "score": 0.39026633205176303, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 5, + "id": "cmt6a4tyt01wn8kc988x5ub9i", + "score": 0.3902046921916232, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 6, + "id": "cmt6a4u0j01wy8kc9yldiydnt", + "score": 0.39014489232730837, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 7, + "id": "cmt6a4u2z01x88kc9vpxgh0zq", + "score": 0.39008685128253223, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 8, + "id": "cmt6a4u4z01xl8kc9detw51zy", + "score": 0.39003049258688005, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + }, + { + "i": 9, + "id": "cmt6a4u6501y28kc9acq00emn", + "score": 0.3899757441396751, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for HTTP request logging levels and even" + } + ] + }, + { + "task": "order-dto-mapping", + "query": "MNEMON benchmark project fact for order DTO mapping and money units", + "goldId": "cmt6a4uau01yq8kc9ara0a13p", + "rankInPage": -1, + "rankInPool": 10, + "pageSize": 10, + "poolSize": 50, + "pageLatencyMs": 39, + "deepLatencyMs": 31, + "goldScore": 0.39050281163664174, + "goldBand": "vector<=1.00", + "top": [ + { + "i": 0, + "id": "cmt6a4uct01z48kc9dm08xdm6", + "score": 0.39125, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 1, + "id": "cmt6a4uew01zh8kc9b980pivm", + "score": 0.3911800806451613, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 2, + "id": "cmt6a4ugx01zt8kc9pszz46u3", + "score": 0.3911123809523809, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 3, + "id": "cmt6a4uj3020a8kc9zbxuak8h", + "score": 0.391046796875, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 4, + "id": "cmt6a4ulm020l8kc9qe3xmp1k", + "score": 0.3909832307692307, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 5, + "id": "cmt6a4unt020y8kc9prz0pzcg", + "score": 0.3909215909090909, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 6, + "id": "cmt6a4uoy021f8kc900ys09mr", + "score": 0.3908617910447762, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 7, + "id": "cmt6a4urv021m8kc9pbjcu2z1", + "score": 0.39080374999999995, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 8, + "id": "cmt6a4uu202248kc9qn2kesdo", + "score": 0.39074739130434777, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + }, + { + "i": 9, + "id": "cmt6a4uw9022f8kc9jcjzuvpq", + "score": 0.39069264285714284, + "band": "vector<=1.00", + "gold": false, + "text": "MNEMON benchmark project fact for order DTO mapping and money units: a" + } + ] + } + ] +} \ No newline at end of file diff --git a/docs/research/memory-formation-query-transform/artifacts/diversity-production-gate/real-embedding-81.json b/docs/research/memory-formation-query-transform/artifacts/diversity-production-gate/real-embedding-81.json new file mode 100644 index 0000000..c4a20ca --- /dev/null +++ b/docs/research/memory-formation-query-transform/artifacts/diversity-production-gate/real-embedding-81.json @@ -0,0 +1,6744 @@ +{ + "generatedAt": "2026-08-24T06:32:59.974Z", + "config": { + "finalLimit": 10, + "baseFlags": { + "RECALL_RERANK_SCALE_FIX": true, + "RECALL_LEXICAL_COVERAGE_FLOOR": true, + "RECALL_RESCUE_SQL_TIEBREAK": true + }, + "diversityOff": {}, + "diversityOn": { + "RECALL_CANDIDATE_POOL_DEPTH": 12, + "RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT": 2, + "jaccardThreshold": 0.9 + }, + "snapshotIsolation": "usage restored before every query and repeat", + "fixtureIntegrity": { + "memoriesChecked": 1210, + "taggedMemories": 1210 + }, + "embeddingModel": "bge-base", + "embeddingDimensions": 768 + }, + "off": { + "label": "diversity-off", + "summary": { + "queryCount": 81, + "precisionAt5": 0.967741935483871, + "recallAt10": 0.9726027397260274, + "mrrAt10": 0.9126200274348422, + "falsePositiveInjectionRate": 0, + "queriesWithFalsePositiveInjection": 0, + "fullPageRate": 0.9125, + "minPageSize": 0, + "meanPageSize": 9.4375, + "latencyP50Ms": 23, + "latencyP95Ms": 31, + "latencyMeanMs": 21.825, + "_counts": { + "mustTop5Count": 62, + "top5HitCount": 60, + "expectedCount": 73, + "relevantAt10Count": 71, + "forbiddenCount": 91, + "forbiddenHitCount": 0 + } + }, + "stabilityChangedQueries": [], + "queries": [ + { + "queryId": "semantic_001", + "category": "semantic", + "expected": [ + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_004_correction" + ], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "resultIds": [ + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_coffee_002", + "alice_cooking_001", + "alice_travel_002", + "alice_worry_001", + "alice_travel_001", + "alice_books_001" + ], + "latencyMs": 45, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "relevantAt10": [ + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_004_correction" + ], + "forbiddenHits": [], + "mrr": 0.6 + }, + { + "queryId": "semantic_002", + "category": "semantic", + "expected": [ + "alice_coffee_002" + ], + "forbidden": [ + "bob_routine_001", + "bob_coffee_001" + ], + "resultIds": [ + "alice_coffee_002", + "alice_calm_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_joy_001", + "alice_health_002", + "alice_coffee_001", + "alice_insight_001", + "alice_coffee_004_correction", + "alice_stress_001" + ], + "latencyMs": 31, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_002" + ], + "relevantAt10": [ + "alice_coffee_002" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_003", + "category": "semantic", + "expected": [ + "alice_work_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_work_001", + "alice_new_preference_001", + "alice_old_preference_001", + "alice_identity_project_001", + "alice_coffee_001", + "alice_yesterday_work_001", + "alice_frustration_001", + "alice_last_week_work_001", + "alice_work_002", + "alice_oldest_memory_001" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [ + "alice_work_001" + ], + "relevantAt10": [ + "alice_work_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_004", + "category": "semantic", + "expected": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "forbidden": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "resultIds": [ + "bob_coffee_001", + "bob_coffee_002", + "bob_routine_001", + "bob_travel_001", + "bob_health_001", + "bob_family_001", + "bob_books_001", + "bob_family_002", + "bob_work_002", + "bob_work_001" + ], + "latencyMs": 24, + "pageSize": 10, + "top5Hits": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "relevantAt10": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "forbiddenHits": [], + "mrr": 0.75 + }, + { + "queryId": "semantic_005", + "category": "semantic", + "expected": [ + "alice_books_001" + ], + "forbidden": [ + "bob_books_001" + ], + "resultIds": [ + "alice_books_001", + "alice_health_002", + "alice_stress_001", + "alice_work_002", + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_travel_001", + "alice_joy_001", + "alice_emotion_change_001" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [ + "alice_books_001" + ], + "relevantAt10": [ + "alice_books_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_006", + "category": "semantic", + "expected": [ + "alice_cooking_001" + ], + "forbidden": [], + "resultIds": [ + "alice_cooking_001", + "alice_travel_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_family_001", + "alice_emotion_change_001", + "alice_joy_001", + "alice_old_preference_001", + "alice_work_001", + "alice_oldest_memory_001" + ], + "latencyMs": 26, + "pageSize": 10, + "top5Hits": [ + "alice_cooking_001" + ], + "relevantAt10": [ + "alice_cooking_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_007", + "category": "semantic", + "expected": [ + "alice_finance_001" + ], + "forbidden": [], + "resultIds": [ + "alice_finance_001", + "alice_worry_001", + "alice_health_002", + "alice_work_002", + "alice_calm_001", + "alice_pride_001", + "alice_insight_001", + "alice_joy_001", + "alice_cooking_001", + "alice_coffee_002" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [ + "alice_finance_001" + ], + "relevantAt10": [ + "alice_finance_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_008", + "category": "semantic", + "expected": [ + "bob_work_001" + ], + "forbidden": [ + "alice_work_001" + ], + "resultIds": [ + "bob_work_001", + "bob_coffee_002", + "bob_travel_001", + "bob_family_001", + "bob_health_001", + "bob_family_002", + "bob_books_001", + "bob_routine_001", + "bob_work_002", + "bob_coffee_001" + ], + "latencyMs": 27, + "pageSize": 10, + "top5Hits": [ + "bob_work_001" + ], + "relevantAt10": [ + "bob_work_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_009", + "category": "semantic", + "expected": [ + "alice_travel_002" + ], + "forbidden": [], + "resultIds": [ + "alice_travel_002", + "alice_travel_001", + "alice_work_003", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_calm_001", + "alice_coffee_004_correction", + "alice_family_002", + "alice_coffee_001", + "alice_work_002" + ], + "latencyMs": 24, + "pageSize": 10, + "top5Hits": [ + "alice_travel_002" + ], + "relevantAt10": [ + "alice_travel_002" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_010", + "category": "semantic", + "expected": [ + "alice_work_003" + ], + "forbidden": [], + "resultIds": [ + "alice_work_003", + "alice_identity_project_001", + "alice_insight_001", + "alice_work_001", + "alice_work_002", + "alice_new_preference_001", + "alice_anxiety_001", + "alice_yesterday_work_001", + "alice_pride_001", + "alice_last_week_work_001" + ], + "latencyMs": 28, + "pageSize": 10, + "top5Hits": [ + "alice_work_003" + ], + "relevantAt10": [ + "alice_work_003" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_011", + "category": "semantic", + "expected": [ + "alice_coffee_004_correction", + "alice_coffee_003_old" + ], + "forbidden": [ + "bob_coffee_001" + ], + "resultIds": [ + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_coffee_001", + "alice_coffee_002", + "alice_calm_001", + "alice_travel_002", + "alice_cooking_001", + "alice_worry_001", + "alice_emotion_change_001", + "alice_work_001" + ], + "latencyMs": 28, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_004_correction" + ], + "relevantAt10": [ + "alice_coffee_004_correction", + "alice_coffee_003_old" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_001", + "category": "emotional", + "expected": [ + "alice_joy_001" + ], + "forbidden": [], + "resultIds": [ + "alice_joy_001", + "alice_calm_001", + "alice_pride_001", + "alice_mixed_emotion_001", + "alice_work_001", + "alice_family_001", + "alice_travel_001", + "alice_family_003", + "alice_books_001", + "alice_finance_001" + ], + "latencyMs": 26, + "pageSize": 10, + "top5Hits": [ + "alice_joy_001" + ], + "relevantAt10": [ + "alice_joy_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_002", + "category": "emotional", + "expected": [ + "alice_grief_001" + ], + "forbidden": [], + "resultIds": [ + "alice_stress_001", + "alice_grief_001", + "alice_anxiety_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_frustration_001", + "alice_family_003", + "alice_work_002", + "alice_coffee_004_correction", + "alice_health_002" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [ + "alice_grief_001" + ], + "relevantAt10": [ + "alice_grief_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "emotional_003", + "category": "emotional", + "expected": [ + "alice_stress_001", + "alice_work_002" + ], + "forbidden": [], + "resultIds": [ + "alice_stress_001", + "alice_anxiety_001", + "alice_grief_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_frustration_001", + "alice_family_003", + "alice_health_002", + "alice_work_002", + "alice_coffee_001" + ], + "latencyMs": 24, + "pageSize": 10, + "top5Hits": [ + "alice_stress_001" + ], + "relevantAt10": [ + "alice_stress_001", + "alice_work_002" + ], + "forbiddenHits": [], + "mrr": 0.5555555555555556 + }, + { + "queryId": "emotional_004", + "category": "emotional", + "expected": [ + "alice_worry_001", + "alice_anxiety_001" + ], + "forbidden": [], + "resultIds": [ + "alice_worry_001", + "alice_anxiety_001", + "alice_stress_001", + "alice_grief_001", + "alice_emotion_change_001", + "alice_frustration_001", + "alice_work_002", + "alice_family_003", + "alice_health_002", + "alice_high_importance_001" + ], + "latencyMs": 24, + "pageSize": 10, + "top5Hits": [ + "alice_worry_001" + ], + "relevantAt10": [ + "alice_worry_001", + "alice_anxiety_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_005", + "category": "emotional", + "expected": [ + "alice_frustration_001" + ], + "forbidden": [], + "resultIds": [ + "alice_frustration_001", + "alice_stress_001", + "alice_anxiety_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_grief_001", + "alice_work_002", + "alice_coffee_004_correction", + "alice_family_003", + "alice_new_preference_001" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [ + "alice_frustration_001" + ], + "relevantAt10": [ + "alice_frustration_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_006", + "category": "emotional", + "expected": [ + "alice_pride_001" + ], + "forbidden": [], + "resultIds": [ + "alice_pride_001", + "alice_joy_001", + "alice_calm_001", + "alice_mixed_emotion_001", + "alice_work_001", + "alice_travel_001", + "alice_health_002", + "alice_books_001", + "alice_family_003", + "alice_oldest_memory_001" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [ + "alice_pride_001" + ], + "relevantAt10": [ + "alice_pride_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_007", + "category": "emotional", + "expected": [ + "alice_stress_001", + "alice_anxiety_001", + "alice_work_002" + ], + "forbidden": [], + "resultIds": [ + "alice_anxiety_001", + "alice_stress_001", + "alice_worry_001", + "alice_emotion_change_001", + "alice_grief_001", + "alice_frustration_001", + "alice_work_002", + "alice_coffee_002", + "alice_family_003", + "alice_coffee_004_correction" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [ + "alice_stress_001" + ], + "relevantAt10": [ + "alice_stress_001", + "alice_anxiety_001", + "alice_work_002" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "emotional_008", + "category": "emotional", + "expected": [ + "alice_mixed_emotion_001" + ], + "forbidden": [], + "resultIds": [ + "alice_mixed_emotion_001", + "alice_joy_001", + "alice_calm_001", + "alice_pride_001", + "alice_finance_001", + "alice_work_001", + "alice_work_002", + "alice_family_003", + "alice_coffee_002", + "alice_health_002" + ], + "latencyMs": 25, + "pageSize": 10, + "top5Hits": [ + "alice_mixed_emotion_001" + ], + "relevantAt10": [ + "alice_mixed_emotion_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_009", + "category": "emotional", + "expected": [ + "alice_emotion_change_001" + ], + "forbidden": [], + "resultIds": [ + "alice_emotion_change_001", + "alice_pride_001", + "alice_calm_001", + "alice_health_002", + "alice_coffee_001", + "alice_insight_001", + "alice_worry_001", + "alice_work_002", + "alice_anxiety_001", + "alice_family_003" + ], + "latencyMs": 25, + "pageSize": 10, + "top5Hits": [ + "alice_emotion_change_001" + ], + "relevantAt10": [ + "alice_emotion_change_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_010", + "category": "emotional", + "expected": [ + "alice_calm_001" + ], + "forbidden": [], + "resultIds": [ + "alice_calm_001", + "alice_anxiety_001", + "alice_joy_001", + "alice_health_002", + "alice_stress_001", + "alice_family_003", + "alice_travel_001", + "alice_emotion_change_001", + "alice_grief_001", + "alice_mixed_emotion_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [ + "alice_calm_001" + ], + "relevantAt10": [ + "alice_calm_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_001", + "category": "temporal", + "expected": [ + "dave_today_001", + "dave_today_002" + ], + "forbidden": [ + "dave_2years_001", + "dave_2years_002" + ], + "resultIds": [ + "dave_today_001", + "dave_today_002" + ], + "latencyMs": 26, + "pageSize": 2, + "top5Hits": [ + "dave_today_001", + "dave_today_002" + ], + "relevantAt10": [ + "dave_today_001", + "dave_today_002" + ], + "forbiddenHits": [], + "mrr": 0.75 + }, + { + "queryId": "temporal_002", + "category": "temporal", + "expected": [ + "dave_today_001" + ], + "forbidden": [ + "dave_6months_001", + "dave_2years_001" + ], + "resultIds": [ + "dave_today_001", + "dave_today_002" + ], + "latencyMs": 18, + "pageSize": 2, + "top5Hits": [ + "dave_today_001" + ], + "relevantAt10": [ + "dave_today_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_003", + "category": "temporal", + "expected": [ + "alice_family_001", + "alice_family_003" + ], + "forbidden": [ + "bob_family_001" + ], + "resultIds": [ + "alice_family_001", + "alice_stress_001", + "alice_family_003", + "alice_work_002", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_recent_convo_001" + ], + "latencyMs": 14, + "pageSize": 7, + "top5Hits": [ + "alice_family_001" + ], + "relevantAt10": [ + "alice_family_001", + "alice_family_003" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_004", + "category": "temporal", + "expected": [ + "alice_last_week_work_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_health_002", + "alice_joy_001", + "alice_work_002", + "alice_stress_001", + "alice_yesterday_work_001", + "alice_last_week_work_001", + "alice_recent_convo_001", + "alice_calm_001", + "alice_worry_001", + "alice_family_003" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [ + "alice_last_week_work_001" + ], + "forbiddenHits": [], + "mrr": 0.16666666666666666 + }, + { + "queryId": "temporal_005", + "category": "temporal", + "expected": [ + "alice_oldest_memory_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_grief_001", + "alice_travel_001", + "alice_stress_001", + "alice_family_002", + "alice_identity_project_001", + "alice_pride_001", + "alice_oldest_memory_001", + "alice_joy_001", + "alice_family_001", + "alice_cooking_001" + ], + "latencyMs": 16, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [ + "alice_oldest_memory_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_006", + "category": "temporal", + "expected": [ + "alice_recent_convo_001", + "alice_yesterday_work_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_work_002", + "alice_recent_convo_001", + "alice_stress_001", + "alice_family_003", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_family_001" + ], + "latencyMs": 21, + "pageSize": 7, + "top5Hits": [ + "alice_recent_convo_001" + ], + "relevantAt10": [ + "alice_recent_convo_001", + "alice_yesterday_work_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "temporal_007", + "category": "temporal", + "expected": [ + "alice_yesterday_work_001" + ], + "forbidden": [], + "resultIds": [ + "alice_yesterday_work_001", + "alice_stress_001", + "alice_work_002", + "alice_recent_convo_001", + "alice_frustration_001", + "alice_family_003", + "alice_family_001" + ], + "latencyMs": 27, + "pageSize": 7, + "top5Hits": [ + "alice_yesterday_work_001" + ], + "relevantAt10": [ + "alice_yesterday_work_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_008", + "category": "temporal", + "expected": [ + "alice_new_preference_001", + "alice_old_preference_001" + ], + "forbidden": [], + "resultIds": [ + "alice_new_preference_001", + "alice_old_preference_001", + "alice_oldest_memory_001", + "alice_work_001", + "alice_insight_001", + "alice_last_week_work_001", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_identity_project_001", + "alice_work_002" + ], + "latencyMs": 31, + "pageSize": 10, + "top5Hits": [ + "alice_new_preference_001" + ], + "relevantAt10": [ + "alice_new_preference_001", + "alice_old_preference_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_009", + "category": "temporal", + "expected": [ + "dave_6months_050" + ], + "forbidden": [ + "dave_today_001" + ], + "resultIds": [], + "latencyMs": 27, + "pageSize": 0, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_010", + "category": "temporal", + "expected": [ + "dave_2years_150" + ], + "forbidden": [ + "dave_today_001" + ], + "resultIds": [], + "latencyMs": 27, + "pageSize": 0, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_011", + "category": "temporal", + "expected": [ + "alice_oldest_memory_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_oldest_memory_001", + "alice_new_preference_001", + "alice_old_preference_001", + "alice_pride_001", + "alice_insight_001", + "alice_identity_project_001", + "alice_health_002", + "alice_work_001", + "alice_stress_001", + "alice_work_002" + ], + "latencyMs": 21, + "pageSize": 10, + "top5Hits": [ + "alice_oldest_memory_001" + ], + "relevantAt10": [ + "alice_oldest_memory_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_001", + "category": "rls_isolation", + "expected": [ + "alice_coffee_001" + ], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002", + "eve_009" + ], + "resultIds": [ + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_003_old", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_low_importance_001", + "alice_travel_001" + ], + "latencyMs": 29, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_001" + ], + "relevantAt10": [ + "alice_coffee_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "rls_002", + "category": "rls_isolation", + "expected": [ + "bob_coffee_001" + ], + "forbidden": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "resultIds": [ + "bob_coffee_001", + "bob_routine_001", + "bob_coffee_002", + "bob_travel_001", + "bob_health_001", + "bob_family_001", + "bob_family_002", + "bob_work_002", + "bob_work_001", + "bob_books_001" + ], + "latencyMs": 27, + "pageSize": 10, + "top5Hits": [ + "bob_coffee_001" + ], + "relevantAt10": [ + "bob_coffee_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_003", + "category": "rls_isolation", + "expected": [ + "alice_family_001" + ], + "forbidden": [ + "bob_family_001", + "bob_family_002" + ], + "resultIds": [ + "alice_family_001", + "alice_cooking_001", + "alice_joy_001", + "alice_family_003", + "alice_finance_001", + "alice_travel_001", + "alice_family_002", + "alice_health_002", + "alice_worry_001", + "alice_grief_001" + ], + "latencyMs": 31, + "pageSize": 10, + "top5Hits": [ + "alice_family_001" + ], + "relevantAt10": [ + "alice_family_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_004", + "category": "rls_isolation", + "expected": [ + "bob_family_001" + ], + "forbidden": [ + "alice_family_001", + "alice_family_002", + "alice_family_003" + ], + "resultIds": [ + "bob_family_001", + "bob_family_002", + "bob_work_002", + "bob_coffee_002", + "bob_travel_001", + "bob_health_001", + "bob_routine_001", + "bob_work_001", + "bob_coffee_001", + "bob_books_001" + ], + "latencyMs": 20, + "pageSize": 10, + "top5Hits": [ + "bob_family_001" + ], + "relevantAt10": [ + "bob_family_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_005", + "category": "rls_isolation", + "expected": [ + "alice_travel_001" + ], + "forbidden": [ + "bob_travel_001" + ], + "resultIds": [ + "alice_travel_001", + "alice_cooking_001", + "alice_travel_002", + "alice_coffee_004_correction", + "alice_family_002", + "alice_joy_001", + "alice_calm_001", + "alice_coffee_003_old", + "alice_family_001", + "alice_coffee_001" + ], + "latencyMs": 31, + "pageSize": 10, + "top5Hits": [ + "alice_travel_001" + ], + "relevantAt10": [ + "alice_travel_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_006", + "category": "rls_isolation", + "expected": [ + "bob_travel_001" + ], + "forbidden": [ + "alice_travel_001" + ], + "resultIds": [ + "bob_travel_001", + "bob_coffee_002", + "bob_health_001", + "bob_coffee_001", + "bob_work_002", + "bob_family_002", + "bob_books_001", + "bob_family_001", + "bob_work_001", + "bob_routine_001" + ], + "latencyMs": 20, + "pageSize": 10, + "top5Hits": [ + "bob_travel_001" + ], + "relevantAt10": [ + "bob_travel_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_007", + "category": "rls_isolation", + "expected": [ + "alice_work_001" + ], + "forbidden": [ + "bob_work_001", + "bob_work_002" + ], + "resultIds": [ + "alice_work_001", + "alice_insight_001", + "alice_new_preference_001", + "alice_oldest_memory_001", + "alice_pride_001", + "alice_work_002", + "alice_old_preference_001", + "alice_identity_project_001", + "alice_anxiety_001", + "alice_yesterday_work_001" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [ + "alice_work_001" + ], + "relevantAt10": [ + "alice_work_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_008", + "category": "rls_isolation", + "expected": [ + "alice_health_001" + ], + "forbidden": [ + "bob_health_001" + ], + "resultIds": [ + "alice_health_001", + "alice_calm_001", + "alice_joy_001", + "alice_new_preference_001", + "alice_health_002", + "alice_cooking_001", + "alice_travel_002", + "alice_anxiety_001", + "alice_travel_001", + "alice_work_002" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [ + "alice_health_001" + ], + "relevantAt10": [ + "alice_health_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_009", + "category": "rls_isolation", + "expected": [ + "bob_health_001" + ], + "forbidden": [ + "alice_health_001", + "alice_health_002" + ], + "resultIds": [ + "bob_health_001", + "bob_family_001", + "bob_travel_001", + "bob_family_002", + "bob_routine_001", + "bob_books_001", + "bob_coffee_002", + "bob_work_002", + "bob_coffee_001", + "bob_work_001" + ], + "latencyMs": 28, + "pageSize": 10, + "top5Hits": [ + "bob_health_001" + ], + "relevantAt10": [ + "bob_health_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_010", + "category": "rls_isolation", + "expected": [ + "bob_routine_001" + ], + "forbidden": [ + "alice_coffee_002" + ], + "resultIds": [ + "bob_routine_001", + "bob_family_001", + "bob_coffee_002", + "bob_coffee_001", + "bob_health_001", + "bob_work_002", + "bob_family_002", + "bob_travel_001", + "bob_work_001", + "bob_books_001" + ], + "latencyMs": 19, + "pageSize": 10, + "top5Hits": [ + "bob_routine_001" + ], + "relevantAt10": [ + "bob_routine_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_001", + "category": "adversarial", + "expected": [], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002", + "bob_routine_001" + ], + "resultIds": [ + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_cooking_001", + "alice_travel_002", + "alice_travel_001", + "alice_emotion_change_001", + "alice_work_001" + ], + "latencyMs": 53, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_002", + "category": "adversarial", + "expected": [], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002", + "bob_family_001" + ], + "resultIds": [ + "alice_high_importance_001", + "alice_health_001", + "alice_pride_001", + "alice_identity_project_001", + "alice_family_001", + "alice_work_003", + "alice_finance_001", + "alice_grief_001", + "alice_address_001", + "alice_phone_001" + ], + "latencyMs": 24, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_003", + "category": "adversarial", + "expected": [], + "forbidden": [ + "carol_unicode_001", + "carol_short_001" + ], + "resultIds": [ + "alice_family_001", + "alice_family_002", + "alice_mixed_emotion_001", + "alice_worry_001", + "alice_coffee_004_correction", + "alice_recent_convo_001", + "alice_work_002", + "alice_identity_project_001", + "alice_family_003", + "alice_travel_002" + ], + "latencyMs": 27, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_004", + "category": "adversarial", + "expected": [], + "forbidden": [ + "bob_family_001", + "bob_family_002" + ], + "resultIds": [ + "alice_mixed_emotion_001", + "alice_family_003", + "alice_grief_001", + "alice_cooking_001", + "alice_joy_001", + "alice_pride_001", + "alice_finance_001", + "alice_travel_001", + "alice_family_001", + "alice_coffee_004_correction" + ], + "latencyMs": 22, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_005", + "category": "adversarial", + "expected": [], + "forbidden": [ + "eve_009", + "eve_003" + ], + "resultIds": [ + "bob_travel_001", + "bob_coffee_002", + "bob_family_002", + "bob_health_001", + "bob_family_001", + "bob_coffee_001", + "bob_books_001", + "bob_routine_001", + "bob_work_002", + "bob_work_001" + ], + "latencyMs": 28, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_001", + "category": "edge_case", + "expected": [ + "eve_003" + ], + "forbidden": [], + "resultIds": [ + "eve_003", + "eve_005", + "eve_007", + "eve_009", + "eve_006", + "eve_008", + "eve_004", + "eve_010", + "eve_002", + "eve_001" + ], + "latencyMs": 17, + "pageSize": 10, + "top5Hits": [ + "eve_003" + ], + "relevantAt10": [ + "eve_003" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_002", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_family_001", + "bob_family_001", + "carol_short_001" + ], + "resultIds": [ + "eve_003", + "eve_005", + "eve_007", + "eve_006", + "eve_004", + "eve_002", + "eve_009", + "eve_010", + "eve_008", + "eve_001" + ], + "latencyMs": 19, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_003", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "bob_coffee_001" + ], + "resultIds": [ + "carol_xss_001", + "carol_xss_003", + "carol_xss_004", + "carol_xss_002", + "carol_short_001", + "carol_sqli_002", + "carol_null_003", + "carol_unicode_002", + "carol_unicode_004", + "carol_special_003" + ], + "latencyMs": 30, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_004", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_coffee_001" + ], + "resultIds": [ + "carol_sqli_001", + "carol_long_001", + "carol_unicode_002", + "carol_unicode_001", + "carol_short_003", + "carol_unicode_005", + "carol_null_003", + "carol_short_001", + "carol_short_002", + "carol_unicode_004" + ], + "latencyMs": 20, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_005", + "category": "edge_case", + "expected": [ + "carol_unicode_001" + ], + "forbidden": [], + "resultIds": [ + "carol_unicode_001", + "carol_unicode_004", + "carol_short_001", + "carol_short_003", + "carol_unicode_002", + "carol_unicode_003", + "carol_null_003", + "carol_short_002", + "carol_xss_002", + "carol_unicode_005" + ], + "latencyMs": 31, + "pageSize": 10, + "top5Hits": [ + "carol_unicode_001" + ], + "relevantAt10": [ + "carol_unicode_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_006", + "category": "edge_case", + "expected": [], + "forbidden": [], + "resultIds": [], + "latencyMs": 0, + "pageSize": 0, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_007", + "category": "edge_case", + "expected": [], + "forbidden": [ + "bob_coffee_001" + ], + "resultIds": [ + "alice_pride_001", + "alice_identity_project_001", + "alice_new_preference_001", + "alice_work_001", + "alice_books_001", + "alice_old_preference_001", + "alice_stress_001", + "alice_travel_001", + "alice_work_003", + "alice_work_002" + ], + "latencyMs": 28, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_008", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "bob_coffee_001" + ], + "resultIds": [ + "carol_unicode_002", + "carol_null_003", + "carol_short_001", + "carol_unicode_003", + "carol_short_003", + "carol_unicode_004", + "carol_xss_002", + "carol_unicode_001", + "carol_xss_001", + "carol_short_002" + ], + "latencyMs": 20, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_009", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "bob_coffee_001" + ], + "resultIds": [ + "carol_sqli_003", + "carol_sqli_002", + "carol_short_001", + "carol_short_003", + "carol_null_003", + "carol_xss_002", + "carol_xss_001", + "carol_special_001", + "carol_short_002", + "carol_unicode_003" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_010", + "category": "edge_case", + "expected": [], + "forbidden": [ + "bob_coffee_001", + "carol_short_001" + ], + "resultIds": [ + "alice_identity_project_001", + "alice_yesterday_work_001", + "alice_books_001", + "alice_recent_convo_001", + "alice_work_003", + "alice_last_week_work_001", + "alice_anxiety_001", + "alice_new_preference_001", + "alice_family_002", + "alice_coffee_003_old" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_011", + "category": "edge_case", + "expected": [], + "forbidden": [], + "resultIds": [ + "alice_pride_001", + "alice_joy_001", + "alice_work_002", + "alice_family_002", + "alice_anxiety_001", + "alice_family_003", + "alice_worry_001", + "alice_finance_001", + "alice_calm_001", + "alice_new_preference_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_012", + "category": "edge_case", + "expected": [ + "alice_coffee_001" + ], + "forbidden": [ + "bob_coffee_001" + ], + "resultIds": [ + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_003_old", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_low_importance_001", + "alice_travel_001" + ], + "latencyMs": 16, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_001" + ], + "relevantAt10": [ + "alice_coffee_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "edge_013", + "category": "edge_case", + "expected": [ + "alice_phone_001" + ], + "forbidden": [], + "resultIds": [ + "alice_phone_001", + "alice_address_001", + "alice_worry_001", + "alice_grief_001", + "alice_work_002", + "alice_anxiety_001", + "alice_family_002", + "alice_joy_001", + "alice_stress_001", + "alice_health_002" + ], + "latencyMs": 26, + "pageSize": 10, + "top5Hits": [ + "alice_phone_001" + ], + "relevantAt10": [ + "alice_phone_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_014", + "category": "edge_case", + "expected": [ + "alice_address_001" + ], + "forbidden": [], + "resultIds": [ + "alice_address_001", + "alice_phone_001", + "alice_joy_001", + "alice_pride_001", + "alice_grief_001", + "alice_worry_001", + "alice_work_002", + "alice_identity_project_001", + "alice_emotion_change_001", + "alice_family_002" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [ + "alice_address_001" + ], + "relevantAt10": [ + "alice_address_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_015", + "category": "edge_case", + "expected": [ + "eve_004" + ], + "forbidden": [ + "alice_work_001", + "bob_work_001" + ], + "resultIds": [ + "eve_005", + "eve_007", + "eve_003", + "eve_004", + "eve_006", + "eve_008", + "eve_002", + "eve_009", + "eve_010", + "eve_001" + ], + "latencyMs": 11, + "pageSize": 10, + "top5Hits": [ + "eve_004" + ], + "relevantAt10": [ + "eve_004" + ], + "forbiddenHits": [], + "mrr": 0.25 + }, + { + "queryId": "cross_001", + "category": "cross_feature", + "expected": [ + "alice_health_001" + ], + "forbidden": [ + "bob_health_001" + ], + "resultIds": [ + "alice_health_001", + "alice_calm_001", + "alice_coffee_002", + "alice_emotion_change_001", + "alice_stress_001", + "alice_coffee_001", + "alice_family_003", + "alice_coffee_003_old", + "alice_cooking_001", + "alice_joy_001" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [ + "alice_health_001" + ], + "relevantAt10": [ + "alice_health_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_002", + "category": "cross_feature", + "expected": [ + "alice_health_002" + ], + "forbidden": [ + "bob_routine_001" + ], + "resultIds": [ + "alice_health_002", + "alice_calm_001", + "alice_joy_001", + "alice_cooking_001", + "alice_insight_001", + "alice_family_003", + "alice_pride_001", + "alice_finance_001", + "alice_high_importance_001", + "alice_travel_001" + ], + "latencyMs": 27, + "pageSize": 10, + "top5Hits": [ + "alice_health_002" + ], + "relevantAt10": [ + "alice_health_002" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_003", + "category": "cross_feature", + "expected": [ + "alice_finance_001" + ], + "forbidden": [], + "resultIds": [ + "alice_finance_001", + "alice_worry_001", + "alice_mixed_emotion_001", + "alice_joy_001", + "alice_work_002", + "alice_calm_001", + "alice_health_002", + "alice_anxiety_001", + "alice_travel_001", + "alice_family_002" + ], + "latencyMs": 16, + "pageSize": 10, + "top5Hits": [ + "alice_finance_001" + ], + "relevantAt10": [ + "alice_finance_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_004", + "category": "cross_feature", + "expected": [ + "alice_family_003" + ], + "forbidden": [ + "bob_family_001" + ], + "resultIds": [ + "alice_family_003", + "alice_joy_001", + "alice_cooking_001", + "alice_family_001", + "alice_mixed_emotion_001", + "alice_calm_001", + "alice_work_002", + "alice_health_002", + "alice_finance_001", + "alice_worry_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [ + "alice_family_003" + ], + "relevantAt10": [ + "alice_family_003" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_005", + "category": "cross_feature", + "expected": [ + "bob_family_001" + ], + "forbidden": [ + "alice_family_003" + ], + "resultIds": [ + "bob_family_001", + "bob_routine_001", + "bob_coffee_002", + "bob_family_002", + "bob_work_002", + "bob_travel_001", + "bob_coffee_001", + "bob_work_001", + "bob_health_001", + "bob_books_001" + ], + "latencyMs": 12, + "pageSize": 10, + "top5Hits": [ + "bob_family_001" + ], + "relevantAt10": [ + "bob_family_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_006", + "category": "cross_feature", + "expected": [ + "alice_identity_project_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_oldest_memory_001", + "alice_identity_project_001", + "alice_anxiety_001", + "alice_pride_001", + "alice_last_week_work_001", + "alice_yesterday_work_001", + "alice_insight_001", + "alice_work_001", + "alice_high_importance_001", + "alice_work_003" + ], + "latencyMs": 16, + "pageSize": 10, + "top5Hits": [ + "alice_identity_project_001" + ], + "relevantAt10": [ + "alice_identity_project_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "cross_007", + "category": "cross_feature", + "expected": [ + "alice_high_importance_001" + ], + "forbidden": [], + "resultIds": [ + "alice_high_importance_001", + "alice_insight_001", + "alice_new_preference_001", + "alice_anxiety_001", + "alice_frustration_001", + "alice_work_001", + "alice_pride_001", + "alice_work_003", + "alice_work_002", + "alice_old_preference_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [ + "alice_high_importance_001" + ], + "relevantAt10": [ + "alice_high_importance_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_008", + "category": "cross_feature", + "expected": [ + "alice_insight_001" + ], + "forbidden": [], + "resultIds": [ + "alice_insight_001", + "alice_emotion_change_001", + "alice_pride_001", + "alice_calm_001", + "alice_anxiety_001", + "alice_coffee_002", + "alice_yesterday_work_001", + "alice_stress_001", + "alice_coffee_001", + "alice_identity_project_001" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [ + "alice_insight_001" + ], + "relevantAt10": [ + "alice_insight_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_009", + "category": "cross_feature", + "expected": [ + "eve_005" + ], + "forbidden": [], + "resultIds": [ + "eve_005", + "eve_007", + "eve_003", + "eve_009", + "eve_004", + "eve_001", + "eve_002", + "eve_008", + "eve_006", + "eve_010" + ], + "latencyMs": 11, + "pageSize": 10, + "top5Hits": [ + "eve_005" + ], + "relevantAt10": [ + "eve_005" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_010", + "category": "cross_feature", + "expected": [ + "eve_007" + ], + "forbidden": [ + "alice_work_001" + ], + "resultIds": [ + "eve_007", + "eve_003", + "eve_005", + "eve_002", + "eve_001", + "eve_010", + "eve_004", + "eve_009", + "eve_006", + "eve_008" + ], + "latencyMs": 12, + "pageSize": 10, + "top5Hits": [ + "eve_007" + ], + "relevantAt10": [ + "eve_007" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_006", + "category": "adversarial", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "alice_family_001", + "alice_work_001" + ], + "resultIds": [ + "bob_work_001", + "bob_family_002", + "bob_books_001", + "bob_coffee_002", + "bob_family_001", + "bob_health_001", + "bob_coffee_001", + "bob_work_002", + "bob_routine_001", + "bob_travel_001" + ], + "latencyMs": 12, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_007", + "category": "adversarial", + "expected": [], + "forbidden": [ + "bob_health_001" + ], + "resultIds": [ + "alice_calm_001", + "alice_joy_001", + "alice_worry_001", + "alice_health_002", + "alice_anxiety_001", + "alice_stress_001", + "alice_coffee_002", + "alice_work_002", + "alice_health_001", + "alice_family_003" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_008", + "category": "adversarial", + "expected": [], + "forbidden": [ + "alice_health_001", + "alice_health_002" + ], + "resultIds": [ + "bob_routine_001", + "bob_family_001", + "bob_health_001", + "bob_family_002", + "bob_coffee_002", + "bob_work_002", + "bob_work_001", + "bob_books_001", + "bob_coffee_001", + "bob_travel_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_009", + "category": "adversarial", + "expected": [], + "forbidden": [ + "carol_unicode_001", + "carol_short_001", + "carol_xss_001" + ], + "resultIds": [ + "eve_005", + "eve_007", + "eve_003", + "eve_001", + "eve_002", + "eve_006", + "eve_004", + "eve_009", + "eve_008", + "eve_010" + ], + "latencyMs": 13, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_010", + "category": "adversarial", + "expected": [], + "forbidden": [ + "dave_today_001", + "dave_today_002" + ], + "resultIds": [ + "alice_joy_001", + "alice_pride_001", + "alice_work_002", + "alice_coffee_004_correction", + "alice_family_002", + "alice_grief_001", + "alice_calm_001", + "alice_emotion_change_001", + "alice_yesterday_work_001", + "alice_travel_001" + ], + "latencyMs": 17, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_016", + "category": "edge_case", + "expected": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "resultIds": [ + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_coffee_002", + "alice_cooking_001", + "alice_travel_002", + "alice_worry_001", + "alice_travel_001", + "alice_books_001" + ], + "latencyMs": 17, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "relevantAt10": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "forbiddenHits": [], + "mrr": 0.6 + }, + { + "queryId": "negative_001", + "category": "semantic", + "expected": [], + "forbidden": [ + "bob_coffee_001", + "carol_short_001", + "eve_001" + ], + "resultIds": [ + "alice_work_001", + "alice_identity_project_001", + "alice_new_preference_001", + "alice_yesterday_work_001", + "alice_coffee_003_old", + "alice_family_001", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_calm_001", + "alice_books_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "negative_002", + "category": "semantic", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "carol_short_001" + ], + "resultIds": [ + "bob_family_001", + "bob_travel_001", + "bob_family_002", + "bob_books_001", + "bob_coffee_001", + "bob_health_001", + "bob_work_001", + "bob_work_002", + "bob_coffee_002", + "bob_routine_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "minimal_001", + "category": "semantic", + "expected": [ + "eve_009" + ], + "forbidden": [], + "resultIds": [ + "eve_003", + "eve_005", + "eve_007", + "eve_009", + "eve_002", + "eve_001", + "eve_004", + "eve_006", + "eve_008", + "eve_010" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [ + "eve_009" + ], + "relevantAt10": [ + "eve_009" + ], + "forbiddenHits": [], + "mrr": 0.25 + } + ], + "repeatedResultIds": { + "semantic_001": [ + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_coffee_002", + "alice_cooking_001", + "alice_travel_002", + "alice_worry_001", + "alice_travel_001", + "alice_books_001" + ], + "semantic_002": [ + "alice_coffee_002", + "alice_calm_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_joy_001", + "alice_health_002", + "alice_coffee_001", + "alice_insight_001", + "alice_coffee_004_correction", + "alice_stress_001" + ], + "semantic_003": [ + "alice_work_001", + "alice_new_preference_001", + "alice_old_preference_001", + "alice_identity_project_001", + "alice_coffee_001", + "alice_yesterday_work_001", + "alice_frustration_001", + "alice_last_week_work_001", + "alice_work_002", + "alice_oldest_memory_001" + ], + "semantic_004": [ + "bob_coffee_001", + "bob_coffee_002", + "bob_routine_001", + "bob_travel_001", + "bob_health_001", + "bob_family_001", + "bob_books_001", + "bob_family_002", + "bob_work_002", + "bob_work_001" + ], + "semantic_005": [ + "alice_books_001", + "alice_health_002", + "alice_stress_001", + "alice_work_002", + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_travel_001", + "alice_joy_001", + "alice_emotion_change_001" + ], + "semantic_006": [ + "alice_cooking_001", + "alice_travel_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_family_001", + "alice_emotion_change_001", + "alice_joy_001", + "alice_old_preference_001", + "alice_work_001", + "alice_oldest_memory_001" + ], + "semantic_007": [ + "alice_finance_001", + "alice_worry_001", + "alice_health_002", + "alice_work_002", + "alice_calm_001", + "alice_pride_001", + "alice_insight_001", + "alice_joy_001", + "alice_cooking_001", + "alice_coffee_002" + ], + "semantic_008": [ + "bob_work_001", + "bob_coffee_002", + "bob_travel_001", + "bob_family_001", + "bob_health_001", + "bob_family_002", + "bob_books_001", + "bob_routine_001", + "bob_work_002", + "bob_coffee_001" + ], + "semantic_009": [ + "alice_travel_002", + "alice_travel_001", + "alice_work_003", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_calm_001", + "alice_coffee_004_correction", + "alice_family_002", + "alice_coffee_001", + "alice_work_002" + ], + "semantic_010": [ + "alice_work_003", + "alice_identity_project_001", + "alice_insight_001", + "alice_work_001", + "alice_work_002", + "alice_new_preference_001", + "alice_anxiety_001", + "alice_yesterday_work_001", + "alice_pride_001", + "alice_last_week_work_001" + ], + "semantic_011": [ + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_coffee_001", + "alice_coffee_002", + "alice_calm_001", + "alice_travel_002", + "alice_cooking_001", + "alice_worry_001", + "alice_emotion_change_001", + "alice_work_001" + ], + "emotional_001": [ + "alice_joy_001", + "alice_calm_001", + "alice_pride_001", + "alice_mixed_emotion_001", + "alice_work_001", + "alice_family_001", + "alice_travel_001", + "alice_family_003", + "alice_books_001", + "alice_finance_001" + ], + "emotional_002": [ + "alice_stress_001", + "alice_grief_001", + "alice_anxiety_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_frustration_001", + "alice_family_003", + "alice_work_002", + "alice_coffee_004_correction", + "alice_health_002" + ], + "emotional_003": [ + "alice_stress_001", + "alice_anxiety_001", + "alice_grief_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_frustration_001", + "alice_family_003", + "alice_health_002", + "alice_work_002", + "alice_coffee_001" + ], + "emotional_004": [ + "alice_worry_001", + "alice_anxiety_001", + "alice_stress_001", + "alice_grief_001", + "alice_emotion_change_001", + "alice_frustration_001", + "alice_work_002", + "alice_family_003", + "alice_health_002", + "alice_high_importance_001" + ], + "emotional_005": [ + "alice_frustration_001", + "alice_stress_001", + "alice_anxiety_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_grief_001", + "alice_work_002", + "alice_coffee_004_correction", + "alice_family_003", + "alice_new_preference_001" + ], + "emotional_006": [ + "alice_pride_001", + "alice_joy_001", + "alice_calm_001", + "alice_mixed_emotion_001", + "alice_work_001", + "alice_travel_001", + "alice_health_002", + "alice_books_001", + "alice_family_003", + "alice_oldest_memory_001" + ], + "emotional_007": [ + "alice_anxiety_001", + "alice_stress_001", + "alice_worry_001", + "alice_emotion_change_001", + "alice_grief_001", + "alice_frustration_001", + "alice_work_002", + "alice_coffee_002", + "alice_family_003", + "alice_coffee_004_correction" + ], + "emotional_008": [ + "alice_mixed_emotion_001", + "alice_joy_001", + "alice_calm_001", + "alice_pride_001", + "alice_finance_001", + "alice_work_001", + "alice_work_002", + "alice_family_003", + "alice_coffee_002", + "alice_health_002" + ], + "emotional_009": [ + "alice_emotion_change_001", + "alice_pride_001", + "alice_calm_001", + "alice_health_002", + "alice_coffee_001", + "alice_insight_001", + "alice_worry_001", + "alice_work_002", + "alice_anxiety_001", + "alice_family_003" + ], + "emotional_010": [ + "alice_calm_001", + "alice_anxiety_001", + "alice_joy_001", + "alice_health_002", + "alice_stress_001", + "alice_family_003", + "alice_travel_001", + "alice_emotion_change_001", + "alice_grief_001", + "alice_mixed_emotion_001" + ], + "temporal_001": [ + "dave_today_001", + "dave_today_002" + ], + "temporal_002": [ + "dave_today_001", + "dave_today_002" + ], + "temporal_003": [ + "alice_family_001", + "alice_stress_001", + "alice_family_003", + "alice_work_002", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_recent_convo_001" + ], + "temporal_004": [ + "alice_health_002", + "alice_joy_001", + "alice_work_002", + "alice_stress_001", + "alice_yesterday_work_001", + "alice_last_week_work_001", + "alice_recent_convo_001", + "alice_calm_001", + "alice_worry_001", + "alice_family_003" + ], + "temporal_005": [ + "alice_grief_001", + "alice_travel_001", + "alice_stress_001", + "alice_family_002", + "alice_identity_project_001", + "alice_pride_001", + "alice_oldest_memory_001", + "alice_joy_001", + "alice_family_001", + "alice_cooking_001" + ], + "temporal_006": [ + "alice_work_002", + "alice_recent_convo_001", + "alice_stress_001", + "alice_family_003", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_family_001" + ], + "temporal_007": [ + "alice_yesterday_work_001", + "alice_stress_001", + "alice_work_002", + "alice_recent_convo_001", + "alice_frustration_001", + "alice_family_003", + "alice_family_001" + ], + "temporal_008": [ + "alice_new_preference_001", + "alice_old_preference_001", + "alice_oldest_memory_001", + "alice_work_001", + "alice_insight_001", + "alice_last_week_work_001", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_identity_project_001", + "alice_work_002" + ], + "temporal_009": [], + "temporal_010": [], + "temporal_011": [ + "alice_oldest_memory_001", + "alice_new_preference_001", + "alice_old_preference_001", + "alice_pride_001", + "alice_insight_001", + "alice_identity_project_001", + "alice_health_002", + "alice_work_001", + "alice_stress_001", + "alice_work_002" + ], + "rls_001": [ + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_003_old", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_low_importance_001", + "alice_travel_001" + ], + "rls_002": [ + "bob_coffee_001", + "bob_routine_001", + "bob_coffee_002", + "bob_travel_001", + "bob_health_001", + "bob_family_001", + "bob_family_002", + "bob_work_002", + "bob_work_001", + "bob_books_001" + ], + "rls_003": [ + "alice_family_001", + "alice_cooking_001", + "alice_joy_001", + "alice_family_003", + "alice_finance_001", + "alice_travel_001", + "alice_family_002", + "alice_health_002", + "alice_worry_001", + "alice_grief_001" + ], + "rls_004": [ + "bob_family_001", + "bob_family_002", + "bob_work_002", + "bob_coffee_002", + "bob_travel_001", + "bob_health_001", + "bob_routine_001", + "bob_work_001", + "bob_coffee_001", + "bob_books_001" + ], + "rls_005": [ + "alice_travel_001", + "alice_cooking_001", + "alice_travel_002", + "alice_coffee_004_correction", + "alice_family_002", + "alice_joy_001", + "alice_calm_001", + "alice_coffee_003_old", + "alice_family_001", + "alice_coffee_001" + ], + "rls_006": [ + "bob_travel_001", + "bob_coffee_002", + "bob_health_001", + "bob_coffee_001", + "bob_work_002", + "bob_family_002", + "bob_books_001", + "bob_family_001", + "bob_work_001", + "bob_routine_001" + ], + "rls_007": [ + "alice_work_001", + "alice_insight_001", + "alice_new_preference_001", + "alice_oldest_memory_001", + "alice_pride_001", + "alice_work_002", + "alice_old_preference_001", + "alice_identity_project_001", + "alice_anxiety_001", + "alice_yesterday_work_001" + ], + "rls_008": [ + "alice_health_001", + "alice_calm_001", + "alice_joy_001", + "alice_new_preference_001", + "alice_health_002", + "alice_cooking_001", + "alice_travel_002", + "alice_anxiety_001", + "alice_travel_001", + "alice_work_002" + ], + "rls_009": [ + "bob_health_001", + "bob_family_001", + "bob_travel_001", + "bob_family_002", + "bob_routine_001", + "bob_books_001", + "bob_coffee_002", + "bob_work_002", + "bob_coffee_001", + "bob_work_001" + ], + "rls_010": [ + "bob_routine_001", + "bob_family_001", + "bob_coffee_002", + "bob_coffee_001", + "bob_health_001", + "bob_work_002", + "bob_family_002", + "bob_travel_001", + "bob_work_001", + "bob_books_001" + ], + "adversarial_001": [ + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_cooking_001", + "alice_travel_002", + "alice_travel_001", + "alice_emotion_change_001", + "alice_work_001" + ], + "adversarial_002": [ + "alice_high_importance_001", + "alice_health_001", + "alice_pride_001", + "alice_identity_project_001", + "alice_family_001", + "alice_work_003", + "alice_finance_001", + "alice_grief_001", + "alice_address_001", + "alice_phone_001" + ], + "adversarial_003": [ + "alice_family_001", + "alice_family_002", + "alice_mixed_emotion_001", + "alice_worry_001", + "alice_coffee_004_correction", + "alice_recent_convo_001", + "alice_work_002", + "alice_identity_project_001", + "alice_family_003", + "alice_travel_002" + ], + "adversarial_004": [ + "alice_mixed_emotion_001", + "alice_family_003", + "alice_grief_001", + "alice_cooking_001", + "alice_joy_001", + "alice_pride_001", + "alice_finance_001", + "alice_travel_001", + "alice_family_001", + "alice_coffee_004_correction" + ], + "adversarial_005": [ + "bob_travel_001", + "bob_coffee_002", + "bob_family_002", + "bob_health_001", + "bob_family_001", + "bob_coffee_001", + "bob_books_001", + "bob_routine_001", + "bob_work_002", + "bob_work_001" + ], + "edge_001": [ + "eve_003", + "eve_005", + "eve_007", + "eve_009", + "eve_006", + "eve_008", + "eve_004", + "eve_010", + "eve_002", + "eve_001" + ], + "edge_002": [ + "eve_003", + "eve_005", + "eve_007", + "eve_006", + "eve_004", + "eve_002", + "eve_009", + "eve_010", + "eve_008", + "eve_001" + ], + "edge_003": [ + "carol_xss_001", + "carol_xss_003", + "carol_xss_004", + "carol_xss_002", + "carol_short_001", + "carol_sqli_002", + "carol_null_003", + "carol_unicode_002", + "carol_unicode_004", + "carol_special_003" + ], + "edge_004": [ + "carol_sqli_001", + "carol_long_001", + "carol_unicode_002", + "carol_unicode_001", + "carol_short_003", + "carol_unicode_005", + "carol_null_003", + "carol_short_001", + "carol_short_002", + "carol_unicode_004" + ], + "edge_005": [ + "carol_unicode_001", + "carol_unicode_004", + "carol_short_001", + "carol_short_003", + "carol_unicode_002", + "carol_unicode_003", + "carol_null_003", + "carol_short_002", + "carol_xss_002", + "carol_unicode_005" + ], + "edge_006": [], + "edge_007": [ + "alice_pride_001", + "alice_identity_project_001", + "alice_new_preference_001", + "alice_work_001", + "alice_books_001", + "alice_old_preference_001", + "alice_stress_001", + "alice_travel_001", + "alice_work_003", + "alice_work_002" + ], + "edge_008": [ + "carol_unicode_002", + "carol_null_003", + "carol_short_001", + "carol_unicode_003", + "carol_short_003", + "carol_unicode_004", + "carol_xss_002", + "carol_unicode_001", + "carol_xss_001", + "carol_short_002" + ], + "edge_009": [ + "carol_sqli_003", + "carol_sqli_002", + "carol_short_001", + "carol_short_003", + "carol_null_003", + "carol_xss_002", + "carol_xss_001", + "carol_special_001", + "carol_short_002", + "carol_unicode_003" + ], + "edge_010": [ + "alice_identity_project_001", + "alice_yesterday_work_001", + "alice_books_001", + "alice_recent_convo_001", + "alice_work_003", + "alice_last_week_work_001", + "alice_anxiety_001", + "alice_new_preference_001", + "alice_family_002", + "alice_coffee_003_old" + ], + "edge_011": [ + "alice_pride_001", + "alice_joy_001", + "alice_work_002", + "alice_family_002", + "alice_anxiety_001", + "alice_family_003", + "alice_worry_001", + "alice_finance_001", + "alice_calm_001", + "alice_new_preference_001" + ], + "edge_012": [ + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_003_old", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_low_importance_001", + "alice_travel_001" + ], + "edge_013": [ + "alice_phone_001", + "alice_address_001", + "alice_worry_001", + "alice_grief_001", + "alice_work_002", + "alice_anxiety_001", + "alice_family_002", + "alice_joy_001", + "alice_stress_001", + "alice_health_002" + ], + "edge_014": [ + "alice_address_001", + "alice_phone_001", + "alice_joy_001", + "alice_pride_001", + "alice_grief_001", + "alice_worry_001", + "alice_work_002", + "alice_identity_project_001", + "alice_emotion_change_001", + "alice_family_002" + ], + "edge_015": [ + "eve_005", + "eve_007", + "eve_003", + "eve_004", + "eve_006", + "eve_008", + "eve_002", + "eve_009", + "eve_010", + "eve_001" + ], + "cross_001": [ + "alice_health_001", + "alice_calm_001", + "alice_coffee_002", + "alice_emotion_change_001", + "alice_stress_001", + "alice_coffee_001", + "alice_family_003", + "alice_coffee_003_old", + "alice_cooking_001", + "alice_joy_001" + ], + "cross_002": [ + "alice_health_002", + "alice_calm_001", + "alice_joy_001", + "alice_cooking_001", + "alice_insight_001", + "alice_family_003", + "alice_pride_001", + "alice_finance_001", + "alice_high_importance_001", + "alice_travel_001" + ], + "cross_003": [ + "alice_finance_001", + "alice_worry_001", + "alice_mixed_emotion_001", + "alice_joy_001", + "alice_work_002", + "alice_calm_001", + "alice_health_002", + "alice_anxiety_001", + "alice_travel_001", + "alice_family_002" + ], + "cross_004": [ + "alice_family_003", + "alice_joy_001", + "alice_cooking_001", + "alice_family_001", + "alice_mixed_emotion_001", + "alice_calm_001", + "alice_work_002", + "alice_health_002", + "alice_finance_001", + "alice_worry_001" + ], + "cross_005": [ + "bob_family_001", + "bob_routine_001", + "bob_coffee_002", + "bob_family_002", + "bob_work_002", + "bob_travel_001", + "bob_coffee_001", + "bob_work_001", + "bob_health_001", + "bob_books_001" + ], + "cross_006": [ + "alice_oldest_memory_001", + "alice_identity_project_001", + "alice_anxiety_001", + "alice_pride_001", + "alice_last_week_work_001", + "alice_yesterday_work_001", + "alice_insight_001", + "alice_work_001", + "alice_high_importance_001", + "alice_work_003" + ], + "cross_007": [ + "alice_high_importance_001", + "alice_insight_001", + "alice_new_preference_001", + "alice_anxiety_001", + "alice_frustration_001", + "alice_work_001", + "alice_pride_001", + "alice_work_003", + "alice_work_002", + "alice_old_preference_001" + ], + "cross_008": [ + "alice_insight_001", + "alice_emotion_change_001", + "alice_pride_001", + "alice_calm_001", + "alice_anxiety_001", + "alice_coffee_002", + "alice_yesterday_work_001", + "alice_stress_001", + "alice_coffee_001", + "alice_identity_project_001" + ], + "cross_009": [ + "eve_005", + "eve_007", + "eve_003", + "eve_009", + "eve_004", + "eve_001", + "eve_002", + "eve_008", + "eve_006", + "eve_010" + ], + "cross_010": [ + "eve_007", + "eve_003", + "eve_005", + "eve_002", + "eve_001", + "eve_010", + "eve_004", + "eve_009", + "eve_006", + "eve_008" + ], + "adversarial_006": [ + "bob_work_001", + "bob_family_002", + "bob_books_001", + "bob_coffee_002", + "bob_family_001", + "bob_health_001", + "bob_coffee_001", + "bob_work_002", + "bob_routine_001", + "bob_travel_001" + ], + "adversarial_007": [ + "alice_calm_001", + "alice_joy_001", + "alice_worry_001", + "alice_health_002", + "alice_anxiety_001", + "alice_stress_001", + "alice_coffee_002", + "alice_work_002", + "alice_health_001", + "alice_family_003" + ], + "adversarial_008": [ + "bob_routine_001", + "bob_family_001", + "bob_health_001", + "bob_family_002", + "bob_coffee_002", + "bob_work_002", + "bob_work_001", + "bob_books_001", + "bob_coffee_001", + "bob_travel_001" + ], + "adversarial_009": [ + "eve_005", + "eve_007", + "eve_003", + "eve_001", + "eve_002", + "eve_006", + "eve_004", + "eve_009", + "eve_008", + "eve_010" + ], + "adversarial_010": [ + "alice_joy_001", + "alice_pride_001", + "alice_work_002", + "alice_coffee_004_correction", + "alice_family_002", + "alice_grief_001", + "alice_calm_001", + "alice_emotion_change_001", + "alice_yesterday_work_001", + "alice_travel_001" + ], + "edge_016": [ + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_coffee_002", + "alice_cooking_001", + "alice_travel_002", + "alice_worry_001", + "alice_travel_001", + "alice_books_001" + ], + "negative_001": [ + "alice_work_001", + "alice_identity_project_001", + "alice_new_preference_001", + "alice_yesterday_work_001", + "alice_coffee_003_old", + "alice_family_001", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_calm_001", + "alice_books_001" + ], + "negative_002": [ + "bob_family_001", + "bob_travel_001", + "bob_family_002", + "bob_books_001", + "bob_coffee_001", + "bob_health_001", + "bob_work_001", + "bob_work_002", + "bob_coffee_002", + "bob_routine_001" + ], + "minimal_001": [ + "eve_003", + "eve_005", + "eve_007", + "eve_009", + "eve_002", + "eve_001", + "eve_004", + "eve_006", + "eve_008", + "eve_010" + ] + } + }, + "on": { + "label": "diversity-on", + "summary": { + "queryCount": 81, + "precisionAt5": 0.967741935483871, + "recallAt10": 0.9726027397260274, + "mrrAt10": 0.9128159905937683, + "falsePositiveInjectionRate": 0, + "queriesWithFalsePositiveInjection": 0, + "fullPageRate": 0.9125, + "minPageSize": 0, + "meanPageSize": 9.4375, + "latencyP50Ms": 18, + "latencyP95Ms": 38, + "latencyMeanMs": 21.4625, + "_counts": { + "mustTop5Count": 62, + "top5HitCount": 60, + "expectedCount": 73, + "relevantAt10Count": 71, + "forbiddenCount": 91, + "forbiddenHitCount": 0 + } + }, + "stabilityChangedQueries": [], + "queries": [ + { + "queryId": "semantic_001", + "category": "semantic", + "expected": [ + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_004_correction" + ], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "resultIds": [ + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_coffee_002", + "alice_cooking_001", + "alice_travel_002", + "alice_worry_001", + "alice_travel_001", + "alice_books_001" + ], + "latencyMs": 27, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "relevantAt10": [ + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_004_correction" + ], + "forbiddenHits": [], + "mrr": 0.6 + }, + { + "queryId": "semantic_002", + "category": "semantic", + "expected": [ + "alice_coffee_002" + ], + "forbidden": [ + "bob_routine_001", + "bob_coffee_001" + ], + "resultIds": [ + "alice_coffee_002", + "alice_calm_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_joy_001", + "alice_health_002", + "alice_coffee_001", + "alice_insight_001", + "alice_coffee_004_correction", + "alice_stress_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_002" + ], + "relevantAt10": [ + "alice_coffee_002" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_003", + "category": "semantic", + "expected": [ + "alice_work_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_work_001", + "alice_new_preference_001", + "alice_old_preference_001", + "alice_identity_project_001", + "alice_coffee_001", + "alice_yesterday_work_001", + "alice_frustration_001", + "alice_last_week_work_001", + "alice_work_002", + "alice_oldest_memory_001" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [ + "alice_work_001" + ], + "relevantAt10": [ + "alice_work_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_004", + "category": "semantic", + "expected": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "forbidden": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "resultIds": [ + "bob_coffee_001", + "bob_coffee_002", + "bob_routine_001", + "bob_travel_001", + "bob_health_001", + "bob_family_001", + "bob_books_001", + "bob_family_002", + "bob_work_002", + "bob_work_001" + ], + "latencyMs": 11, + "pageSize": 10, + "top5Hits": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "relevantAt10": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "forbiddenHits": [], + "mrr": 0.75 + }, + { + "queryId": "semantic_005", + "category": "semantic", + "expected": [ + "alice_books_001" + ], + "forbidden": [ + "bob_books_001" + ], + "resultIds": [ + "alice_books_001", + "alice_health_002", + "alice_stress_001", + "alice_work_002", + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_travel_001", + "alice_joy_001", + "alice_emotion_change_001" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [ + "alice_books_001" + ], + "relevantAt10": [ + "alice_books_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_006", + "category": "semantic", + "expected": [ + "alice_cooking_001" + ], + "forbidden": [], + "resultIds": [ + "alice_cooking_001", + "alice_travel_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_family_001", + "alice_emotion_change_001", + "alice_joy_001", + "alice_old_preference_001", + "alice_work_001", + "alice_oldest_memory_001" + ], + "latencyMs": 13, + "pageSize": 10, + "top5Hits": [ + "alice_cooking_001" + ], + "relevantAt10": [ + "alice_cooking_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_007", + "category": "semantic", + "expected": [ + "alice_finance_001" + ], + "forbidden": [], + "resultIds": [ + "alice_finance_001", + "alice_worry_001", + "alice_health_002", + "alice_work_002", + "alice_calm_001", + "alice_pride_001", + "alice_insight_001", + "alice_joy_001", + "alice_cooking_001", + "alice_coffee_002" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [ + "alice_finance_001" + ], + "relevantAt10": [ + "alice_finance_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_008", + "category": "semantic", + "expected": [ + "bob_work_001" + ], + "forbidden": [ + "alice_work_001" + ], + "resultIds": [ + "bob_work_001", + "bob_coffee_002", + "bob_travel_001", + "bob_family_001", + "bob_health_001", + "bob_family_002", + "bob_books_001", + "bob_routine_001", + "bob_work_002", + "bob_coffee_001" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [ + "bob_work_001" + ], + "relevantAt10": [ + "bob_work_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_009", + "category": "semantic", + "expected": [ + "alice_travel_002" + ], + "forbidden": [], + "resultIds": [ + "alice_travel_002", + "alice_travel_001", + "alice_work_003", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_calm_001", + "alice_coffee_004_correction", + "alice_family_002", + "alice_coffee_001", + "alice_work_002" + ], + "latencyMs": 28, + "pageSize": 10, + "top5Hits": [ + "alice_travel_002" + ], + "relevantAt10": [ + "alice_travel_002" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_010", + "category": "semantic", + "expected": [ + "alice_work_003" + ], + "forbidden": [], + "resultIds": [ + "alice_work_003", + "alice_identity_project_001", + "alice_insight_001", + "alice_work_001", + "alice_work_002", + "alice_new_preference_001", + "alice_anxiety_001", + "alice_yesterday_work_001", + "alice_pride_001", + "alice_last_week_work_001" + ], + "latencyMs": 19, + "pageSize": 10, + "top5Hits": [ + "alice_work_003" + ], + "relevantAt10": [ + "alice_work_003" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "semantic_011", + "category": "semantic", + "expected": [ + "alice_coffee_004_correction", + "alice_coffee_003_old" + ], + "forbidden": [ + "bob_coffee_001" + ], + "resultIds": [ + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_coffee_001", + "alice_coffee_002", + "alice_calm_001", + "alice_travel_002", + "alice_cooking_001", + "alice_worry_001", + "alice_emotion_change_001", + "alice_work_001" + ], + "latencyMs": 17, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_004_correction" + ], + "relevantAt10": [ + "alice_coffee_004_correction", + "alice_coffee_003_old" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_001", + "category": "emotional", + "expected": [ + "alice_joy_001" + ], + "forbidden": [], + "resultIds": [ + "alice_joy_001", + "alice_calm_001", + "alice_pride_001", + "alice_mixed_emotion_001", + "alice_family_001", + "alice_travel_001", + "alice_family_003", + "alice_low_importance_001", + "alice_emotion_change_001", + "alice_worry_001" + ], + "latencyMs": 17, + "pageSize": 10, + "top5Hits": [ + "alice_joy_001" + ], + "relevantAt10": [ + "alice_joy_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_002", + "category": "emotional", + "expected": [ + "alice_grief_001" + ], + "forbidden": [], + "resultIds": [ + "alice_stress_001", + "alice_grief_001", + "alice_anxiety_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_family_003", + "alice_work_002", + "alice_coffee_004_correction", + "alice_health_002", + "alice_calm_001" + ], + "latencyMs": 19, + "pageSize": 10, + "top5Hits": [ + "alice_grief_001" + ], + "relevantAt10": [ + "alice_grief_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "emotional_003", + "category": "emotional", + "expected": [ + "alice_stress_001", + "alice_work_002" + ], + "forbidden": [], + "resultIds": [ + "alice_stress_001", + "alice_anxiety_001", + "alice_grief_001", + "alice_emotion_change_001", + "alice_family_003", + "alice_health_002", + "alice_work_002", + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_calm_001" + ], + "latencyMs": 16, + "pageSize": 10, + "top5Hits": [ + "alice_stress_001" + ], + "relevantAt10": [ + "alice_stress_001", + "alice_work_002" + ], + "forbiddenHits": [], + "mrr": 0.5714285714285714 + }, + { + "queryId": "emotional_004", + "category": "emotional", + "expected": [ + "alice_worry_001", + "alice_anxiety_001" + ], + "forbidden": [], + "resultIds": [ + "alice_worry_001", + "alice_anxiety_001", + "alice_stress_001", + "alice_grief_001", + "alice_work_002", + "alice_family_003", + "alice_health_002", + "alice_high_importance_001", + "alice_coffee_001", + "alice_mixed_emotion_001" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [ + "alice_worry_001" + ], + "relevantAt10": [ + "alice_worry_001", + "alice_anxiety_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_005", + "category": "emotional", + "expected": [ + "alice_frustration_001" + ], + "forbidden": [], + "resultIds": [ + "alice_frustration_001", + "alice_stress_001", + "alice_anxiety_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_work_002", + "alice_coffee_004_correction", + "alice_family_003", + "alice_new_preference_001", + "alice_coffee_002" + ], + "latencyMs": 20, + "pageSize": 10, + "top5Hits": [ + "alice_frustration_001" + ], + "relevantAt10": [ + "alice_frustration_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_006", + "category": "emotional", + "expected": [ + "alice_pride_001" + ], + "forbidden": [], + "resultIds": [ + "alice_pride_001", + "alice_joy_001", + "alice_travel_001", + "alice_health_002", + "alice_books_001", + "alice_family_003", + "alice_oldest_memory_001", + "alice_work_002", + "alice_emotion_change_001", + "alice_grief_001" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [ + "alice_pride_001" + ], + "relevantAt10": [ + "alice_pride_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_007", + "category": "emotional", + "expected": [ + "alice_stress_001", + "alice_anxiety_001", + "alice_work_002" + ], + "forbidden": [], + "resultIds": [ + "alice_anxiety_001", + "alice_stress_001", + "alice_worry_001", + "alice_emotion_change_001", + "alice_grief_001", + "alice_work_002", + "alice_coffee_002", + "alice_family_003", + "alice_coffee_004_correction", + "alice_calm_001" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [ + "alice_stress_001" + ], + "relevantAt10": [ + "alice_stress_001", + "alice_anxiety_001", + "alice_work_002" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "emotional_008", + "category": "emotional", + "expected": [ + "alice_mixed_emotion_001" + ], + "forbidden": [], + "resultIds": [ + "alice_mixed_emotion_001", + "alice_joy_001", + "alice_calm_001", + "alice_pride_001", + "alice_finance_001", + "alice_work_002", + "alice_family_003", + "alice_coffee_002", + "alice_worry_001", + "alice_emotion_change_001" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [ + "alice_mixed_emotion_001" + ], + "relevantAt10": [ + "alice_mixed_emotion_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_009", + "category": "emotional", + "expected": [ + "alice_emotion_change_001" + ], + "forbidden": [], + "resultIds": [ + "alice_emotion_change_001", + "alice_pride_001", + "alice_calm_001", + "alice_health_002", + "alice_coffee_001", + "alice_insight_001", + "alice_worry_001", + "alice_work_002", + "alice_anxiety_001", + "alice_family_003" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [ + "alice_emotion_change_001" + ], + "relevantAt10": [ + "alice_emotion_change_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "emotional_010", + "category": "emotional", + "expected": [ + "alice_calm_001" + ], + "forbidden": [], + "resultIds": [ + "alice_calm_001", + "alice_anxiety_001", + "alice_joy_001", + "alice_health_002", + "alice_stress_001", + "alice_family_003", + "alice_travel_001", + "alice_emotion_change_001", + "alice_grief_001", + "alice_mixed_emotion_001" + ], + "latencyMs": 17, + "pageSize": 10, + "top5Hits": [ + "alice_calm_001" + ], + "relevantAt10": [ + "alice_calm_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_001", + "category": "temporal", + "expected": [ + "dave_today_001", + "dave_today_002" + ], + "forbidden": [ + "dave_2years_001", + "dave_2years_002" + ], + "resultIds": [ + "dave_today_001", + "dave_today_002" + ], + "latencyMs": 19, + "pageSize": 2, + "top5Hits": [ + "dave_today_001", + "dave_today_002" + ], + "relevantAt10": [ + "dave_today_001", + "dave_today_002" + ], + "forbiddenHits": [], + "mrr": 0.75 + }, + { + "queryId": "temporal_002", + "category": "temporal", + "expected": [ + "dave_today_001" + ], + "forbidden": [ + "dave_6months_001", + "dave_2years_001" + ], + "resultIds": [ + "dave_today_001", + "dave_today_002" + ], + "latencyMs": 18, + "pageSize": 2, + "top5Hits": [ + "dave_today_001" + ], + "relevantAt10": [ + "dave_today_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_003", + "category": "temporal", + "expected": [ + "alice_family_001", + "alice_family_003" + ], + "forbidden": [ + "bob_family_001" + ], + "resultIds": [ + "alice_family_001", + "alice_stress_001", + "alice_family_003", + "alice_work_002", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_recent_convo_001" + ], + "latencyMs": 16, + "pageSize": 7, + "top5Hits": [ + "alice_family_001" + ], + "relevantAt10": [ + "alice_family_001", + "alice_family_003" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_004", + "category": "temporal", + "expected": [ + "alice_last_week_work_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_health_002", + "alice_joy_001", + "alice_work_002", + "alice_stress_001", + "alice_yesterday_work_001", + "alice_last_week_work_001", + "alice_recent_convo_001", + "alice_calm_001", + "alice_worry_001", + "alice_family_003" + ], + "latencyMs": 16, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [ + "alice_last_week_work_001" + ], + "forbiddenHits": [], + "mrr": 0.16666666666666666 + }, + { + "queryId": "temporal_005", + "category": "temporal", + "expected": [ + "alice_oldest_memory_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_grief_001", + "alice_travel_001", + "alice_stress_001", + "alice_family_002", + "alice_identity_project_001", + "alice_pride_001", + "alice_oldest_memory_001", + "alice_joy_001", + "alice_family_001", + "alice_cooking_001" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [ + "alice_oldest_memory_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_006", + "category": "temporal", + "expected": [ + "alice_recent_convo_001", + "alice_yesterday_work_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_work_002", + "alice_recent_convo_001", + "alice_stress_001", + "alice_family_003", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_family_001" + ], + "latencyMs": 15, + "pageSize": 7, + "top5Hits": [ + "alice_recent_convo_001" + ], + "relevantAt10": [ + "alice_recent_convo_001", + "alice_yesterday_work_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "temporal_007", + "category": "temporal", + "expected": [ + "alice_yesterday_work_001" + ], + "forbidden": [], + "resultIds": [ + "alice_yesterday_work_001", + "alice_stress_001", + "alice_work_002", + "alice_recent_convo_001", + "alice_frustration_001", + "alice_family_003", + "alice_family_001" + ], + "latencyMs": 18, + "pageSize": 7, + "top5Hits": [ + "alice_yesterday_work_001" + ], + "relevantAt10": [ + "alice_yesterday_work_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_008", + "category": "temporal", + "expected": [ + "alice_new_preference_001", + "alice_old_preference_001" + ], + "forbidden": [], + "resultIds": [ + "alice_new_preference_001", + "alice_old_preference_001", + "alice_oldest_memory_001", + "alice_work_001", + "alice_insight_001", + "alice_last_week_work_001", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_identity_project_001", + "alice_work_002" + ], + "latencyMs": 21, + "pageSize": 10, + "top5Hits": [ + "alice_new_preference_001" + ], + "relevantAt10": [ + "alice_new_preference_001", + "alice_old_preference_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_009", + "category": "temporal", + "expected": [ + "dave_6months_050" + ], + "forbidden": [ + "dave_today_001" + ], + "resultIds": [], + "latencyMs": 19, + "pageSize": 0, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_010", + "category": "temporal", + "expected": [ + "dave_2years_150" + ], + "forbidden": [ + "dave_today_001" + ], + "resultIds": [], + "latencyMs": 17, + "pageSize": 0, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "temporal_011", + "category": "temporal", + "expected": [ + "alice_oldest_memory_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_oldest_memory_001", + "alice_new_preference_001", + "alice_old_preference_001", + "alice_pride_001", + "alice_insight_001", + "alice_identity_project_001", + "alice_health_002", + "alice_work_001", + "alice_stress_001", + "alice_work_002" + ], + "latencyMs": 24, + "pageSize": 10, + "top5Hits": [ + "alice_oldest_memory_001" + ], + "relevantAt10": [ + "alice_oldest_memory_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_001", + "category": "rls_isolation", + "expected": [ + "alice_coffee_001" + ], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002", + "eve_009" + ], + "resultIds": [ + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_003_old", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_low_importance_001", + "alice_travel_001" + ], + "latencyMs": 32, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_001" + ], + "relevantAt10": [ + "alice_coffee_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "rls_002", + "category": "rls_isolation", + "expected": [ + "bob_coffee_001" + ], + "forbidden": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "resultIds": [ + "bob_coffee_001", + "bob_routine_001", + "bob_coffee_002", + "bob_travel_001", + "bob_health_001", + "bob_family_001", + "bob_family_002", + "bob_work_002", + "bob_work_001", + "bob_books_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [ + "bob_coffee_001" + ], + "relevantAt10": [ + "bob_coffee_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_003", + "category": "rls_isolation", + "expected": [ + "alice_family_001" + ], + "forbidden": [ + "bob_family_001", + "bob_family_002" + ], + "resultIds": [ + "alice_family_001", + "alice_cooking_001", + "alice_joy_001", + "alice_family_003", + "alice_finance_001", + "alice_travel_001", + "alice_family_002", + "alice_health_002", + "alice_worry_001", + "alice_grief_001" + ], + "latencyMs": 17, + "pageSize": 10, + "top5Hits": [ + "alice_family_001" + ], + "relevantAt10": [ + "alice_family_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_004", + "category": "rls_isolation", + "expected": [ + "bob_family_001" + ], + "forbidden": [ + "alice_family_001", + "alice_family_002", + "alice_family_003" + ], + "resultIds": [ + "bob_family_001", + "bob_family_002", + "bob_work_002", + "bob_coffee_002", + "bob_travel_001", + "bob_health_001", + "bob_routine_001", + "bob_work_001", + "bob_coffee_001", + "bob_books_001" + ], + "latencyMs": 16, + "pageSize": 10, + "top5Hits": [ + "bob_family_001" + ], + "relevantAt10": [ + "bob_family_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_005", + "category": "rls_isolation", + "expected": [ + "alice_travel_001" + ], + "forbidden": [ + "bob_travel_001" + ], + "resultIds": [ + "alice_travel_001", + "alice_cooking_001", + "alice_travel_002", + "alice_coffee_004_correction", + "alice_family_002", + "alice_joy_001", + "alice_calm_001", + "alice_coffee_003_old", + "alice_family_001", + "alice_coffee_001" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [ + "alice_travel_001" + ], + "relevantAt10": [ + "alice_travel_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_006", + "category": "rls_isolation", + "expected": [ + "bob_travel_001" + ], + "forbidden": [ + "alice_travel_001" + ], + "resultIds": [ + "bob_travel_001", + "bob_coffee_002", + "bob_health_001", + "bob_coffee_001", + "bob_work_002", + "bob_family_002", + "bob_books_001", + "bob_family_001", + "bob_work_001", + "bob_routine_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [ + "bob_travel_001" + ], + "relevantAt10": [ + "bob_travel_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_007", + "category": "rls_isolation", + "expected": [ + "alice_work_001" + ], + "forbidden": [ + "bob_work_001", + "bob_work_002" + ], + "resultIds": [ + "alice_work_001", + "alice_insight_001", + "alice_new_preference_001", + "alice_oldest_memory_001", + "alice_pride_001", + "alice_work_002", + "alice_old_preference_001", + "alice_identity_project_001", + "alice_anxiety_001", + "alice_yesterday_work_001" + ], + "latencyMs": 26, + "pageSize": 10, + "top5Hits": [ + "alice_work_001" + ], + "relevantAt10": [ + "alice_work_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_008", + "category": "rls_isolation", + "expected": [ + "alice_health_001" + ], + "forbidden": [ + "bob_health_001" + ], + "resultIds": [ + "alice_health_001", + "alice_calm_001", + "alice_joy_001", + "alice_new_preference_001", + "alice_health_002", + "alice_cooking_001", + "alice_travel_002", + "alice_anxiety_001", + "alice_travel_001", + "alice_work_002" + ], + "latencyMs": 20, + "pageSize": 10, + "top5Hits": [ + "alice_health_001" + ], + "relevantAt10": [ + "alice_health_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_009", + "category": "rls_isolation", + "expected": [ + "bob_health_001" + ], + "forbidden": [ + "alice_health_001", + "alice_health_002" + ], + "resultIds": [ + "bob_health_001", + "bob_family_001", + "bob_travel_001", + "bob_family_002", + "bob_routine_001", + "bob_books_001", + "bob_coffee_002", + "bob_work_002", + "bob_coffee_001", + "bob_work_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [ + "bob_health_001" + ], + "relevantAt10": [ + "bob_health_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "rls_010", + "category": "rls_isolation", + "expected": [ + "bob_routine_001" + ], + "forbidden": [ + "alice_coffee_002" + ], + "resultIds": [ + "bob_routine_001", + "bob_family_001", + "bob_coffee_002", + "bob_coffee_001", + "bob_health_001", + "bob_work_002", + "bob_family_002", + "bob_travel_001", + "bob_work_001", + "bob_books_001" + ], + "latencyMs": 16, + "pageSize": 10, + "top5Hits": [ + "bob_routine_001" + ], + "relevantAt10": [ + "bob_routine_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_001", + "category": "adversarial", + "expected": [], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002", + "bob_routine_001" + ], + "resultIds": [ + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_cooking_001", + "alice_travel_002", + "alice_travel_001", + "alice_emotion_change_001", + "alice_work_001" + ], + "latencyMs": 19, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_002", + "category": "adversarial", + "expected": [], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002", + "bob_family_001" + ], + "resultIds": [ + "alice_high_importance_001", + "alice_health_001", + "alice_pride_001", + "alice_identity_project_001", + "alice_family_001", + "alice_work_003", + "alice_finance_001", + "alice_grief_001", + "alice_address_001", + "alice_phone_001" + ], + "latencyMs": 19, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_003", + "category": "adversarial", + "expected": [], + "forbidden": [ + "carol_unicode_001", + "carol_short_001" + ], + "resultIds": [ + "alice_family_001", + "alice_family_002", + "alice_mixed_emotion_001", + "alice_worry_001", + "alice_coffee_004_correction", + "alice_recent_convo_001", + "alice_work_002", + "alice_identity_project_001", + "alice_family_003", + "alice_travel_002" + ], + "latencyMs": 19, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_004", + "category": "adversarial", + "expected": [], + "forbidden": [ + "bob_family_001", + "bob_family_002" + ], + "resultIds": [ + "alice_mixed_emotion_001", + "alice_family_003", + "alice_grief_001", + "alice_cooking_001", + "alice_joy_001", + "alice_pride_001", + "alice_finance_001", + "alice_travel_001", + "alice_family_001", + "alice_coffee_004_correction" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_005", + "category": "adversarial", + "expected": [], + "forbidden": [ + "eve_009", + "eve_003" + ], + "resultIds": [ + "bob_travel_001", + "bob_coffee_002", + "bob_family_002", + "bob_health_001", + "bob_family_001", + "bob_coffee_001", + "bob_books_001", + "bob_routine_001", + "bob_work_002", + "bob_work_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_001", + "category": "edge_case", + "expected": [ + "eve_003" + ], + "forbidden": [], + "resultIds": [ + "eve_003", + "eve_005", + "eve_007", + "eve_009", + "eve_006", + "eve_008", + "eve_004", + "eve_010", + "eve_002", + "eve_001" + ], + "latencyMs": 12, + "pageSize": 10, + "top5Hits": [ + "eve_003" + ], + "relevantAt10": [ + "eve_003" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_002", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_family_001", + "bob_family_001", + "carol_short_001" + ], + "resultIds": [ + "eve_003", + "eve_005", + "eve_007", + "eve_006", + "eve_004", + "eve_002", + "eve_009", + "eve_010", + "eve_008", + "eve_001" + ], + "latencyMs": 12, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_003", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "bob_coffee_001" + ], + "resultIds": [ + "carol_xss_001", + "carol_xss_003", + "carol_xss_004", + "carol_xss_002", + "carol_short_001", + "carol_sqli_002", + "carol_null_003", + "carol_unicode_002", + "carol_unicode_004", + "carol_special_003" + ], + "latencyMs": 16, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_004", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_coffee_001" + ], + "resultIds": [ + "carol_sqli_001", + "carol_long_001", + "carol_unicode_002", + "carol_unicode_001", + "carol_short_003", + "carol_unicode_005", + "carol_null_003", + "carol_short_001", + "carol_short_002", + "carol_unicode_004" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_005", + "category": "edge_case", + "expected": [ + "carol_unicode_001" + ], + "forbidden": [], + "resultIds": [ + "carol_unicode_001", + "carol_unicode_004", + "carol_short_001", + "carol_short_003", + "carol_unicode_002", + "carol_unicode_003", + "carol_null_003", + "carol_short_002", + "carol_xss_002", + "carol_unicode_005" + ], + "latencyMs": 14, + "pageSize": 10, + "top5Hits": [ + "carol_unicode_001" + ], + "relevantAt10": [ + "carol_unicode_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_006", + "category": "edge_case", + "expected": [], + "forbidden": [], + "resultIds": [], + "latencyMs": 0, + "pageSize": 0, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_007", + "category": "edge_case", + "expected": [], + "forbidden": [ + "bob_coffee_001" + ], + "resultIds": [ + "alice_pride_001", + "alice_identity_project_001", + "alice_new_preference_001", + "alice_work_001", + "alice_books_001", + "alice_old_preference_001", + "alice_stress_001", + "alice_travel_001", + "alice_work_003", + "alice_work_002" + ], + "latencyMs": 20, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_008", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "bob_coffee_001" + ], + "resultIds": [ + "carol_unicode_002", + "carol_null_003", + "carol_short_001", + "carol_unicode_003", + "carol_short_003", + "carol_unicode_004", + "carol_xss_002", + "carol_unicode_001", + "carol_xss_001", + "carol_short_002" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_009", + "category": "edge_case", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "bob_coffee_001" + ], + "resultIds": [ + "carol_sqli_003", + "carol_sqli_002", + "carol_short_001", + "carol_short_003", + "carol_null_003", + "carol_xss_002", + "carol_xss_001", + "carol_special_001", + "carol_short_002", + "carol_unicode_003" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_010", + "category": "edge_case", + "expected": [], + "forbidden": [ + "bob_coffee_001", + "carol_short_001" + ], + "resultIds": [ + "alice_identity_project_001", + "alice_yesterday_work_001", + "alice_books_001", + "alice_recent_convo_001", + "alice_work_003", + "alice_last_week_work_001", + "alice_anxiety_001", + "alice_new_preference_001", + "alice_family_002", + "alice_coffee_003_old" + ], + "latencyMs": 17, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_011", + "category": "edge_case", + "expected": [], + "forbidden": [], + "resultIds": [ + "alice_pride_001", + "alice_joy_001", + "alice_work_002", + "alice_family_002", + "alice_anxiety_001", + "alice_family_003", + "alice_worry_001", + "alice_finance_001", + "alice_calm_001", + "alice_new_preference_001" + ], + "latencyMs": 15, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_012", + "category": "edge_case", + "expected": [ + "alice_coffee_001" + ], + "forbidden": [ + "bob_coffee_001" + ], + "resultIds": [ + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_003_old", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_low_importance_001", + "alice_travel_001" + ], + "latencyMs": 17, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_001" + ], + "relevantAt10": [ + "alice_coffee_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "edge_013", + "category": "edge_case", + "expected": [ + "alice_phone_001" + ], + "forbidden": [], + "resultIds": [ + "alice_phone_001", + "alice_address_001", + "alice_worry_001", + "alice_grief_001", + "alice_work_002", + "alice_anxiety_001", + "alice_family_002", + "alice_joy_001", + "alice_stress_001", + "alice_health_002" + ], + "latencyMs": 21, + "pageSize": 10, + "top5Hits": [ + "alice_phone_001" + ], + "relevantAt10": [ + "alice_phone_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_014", + "category": "edge_case", + "expected": [ + "alice_address_001" + ], + "forbidden": [], + "resultIds": [ + "alice_address_001", + "alice_phone_001", + "alice_joy_001", + "alice_pride_001", + "alice_grief_001", + "alice_worry_001", + "alice_work_002", + "alice_identity_project_001", + "alice_emotion_change_001", + "alice_family_002" + ], + "latencyMs": 36, + "pageSize": 10, + "top5Hits": [ + "alice_address_001" + ], + "relevantAt10": [ + "alice_address_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_015", + "category": "edge_case", + "expected": [ + "eve_004" + ], + "forbidden": [ + "alice_work_001", + "bob_work_001" + ], + "resultIds": [ + "eve_005", + "eve_007", + "eve_003", + "eve_004", + "eve_006", + "eve_008", + "eve_002", + "eve_009", + "eve_010", + "eve_001" + ], + "latencyMs": 27, + "pageSize": 10, + "top5Hits": [ + "eve_004" + ], + "relevantAt10": [ + "eve_004" + ], + "forbiddenHits": [], + "mrr": 0.25 + }, + { + "queryId": "cross_001", + "category": "cross_feature", + "expected": [ + "alice_health_001" + ], + "forbidden": [ + "bob_health_001" + ], + "resultIds": [ + "alice_health_001", + "alice_calm_001", + "alice_coffee_002", + "alice_emotion_change_001", + "alice_stress_001", + "alice_coffee_001", + "alice_family_003", + "alice_coffee_003_old", + "alice_cooking_001", + "alice_joy_001" + ], + "latencyMs": 29, + "pageSize": 10, + "top5Hits": [ + "alice_health_001" + ], + "relevantAt10": [ + "alice_health_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_002", + "category": "cross_feature", + "expected": [ + "alice_health_002" + ], + "forbidden": [ + "bob_routine_001" + ], + "resultIds": [ + "alice_health_002", + "alice_calm_001", + "alice_joy_001", + "alice_cooking_001", + "alice_insight_001", + "alice_family_003", + "alice_pride_001", + "alice_finance_001", + "alice_high_importance_001", + "alice_travel_001" + ], + "latencyMs": 29, + "pageSize": 10, + "top5Hits": [ + "alice_health_002" + ], + "relevantAt10": [ + "alice_health_002" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_003", + "category": "cross_feature", + "expected": [ + "alice_finance_001" + ], + "forbidden": [], + "resultIds": [ + "alice_finance_001", + "alice_worry_001", + "alice_mixed_emotion_001", + "alice_joy_001", + "alice_work_002", + "alice_calm_001", + "alice_health_002", + "alice_anxiety_001", + "alice_travel_001", + "alice_family_002" + ], + "latencyMs": 36, + "pageSize": 10, + "top5Hits": [ + "alice_finance_001" + ], + "relevantAt10": [ + "alice_finance_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_004", + "category": "cross_feature", + "expected": [ + "alice_family_003" + ], + "forbidden": [ + "bob_family_001" + ], + "resultIds": [ + "alice_family_003", + "alice_joy_001", + "alice_cooking_001", + "alice_family_001", + "alice_mixed_emotion_001", + "alice_calm_001", + "alice_work_002", + "alice_health_002", + "alice_finance_001", + "alice_worry_001" + ], + "latencyMs": 37, + "pageSize": 10, + "top5Hits": [ + "alice_family_003" + ], + "relevantAt10": [ + "alice_family_003" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_005", + "category": "cross_feature", + "expected": [ + "bob_family_001" + ], + "forbidden": [ + "alice_family_003" + ], + "resultIds": [ + "bob_family_001", + "bob_routine_001", + "bob_coffee_002", + "bob_family_002", + "bob_work_002", + "bob_travel_001", + "bob_coffee_001", + "bob_work_001", + "bob_health_001", + "bob_books_001" + ], + "latencyMs": 33, + "pageSize": 10, + "top5Hits": [ + "bob_family_001" + ], + "relevantAt10": [ + "bob_family_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_006", + "category": "cross_feature", + "expected": [ + "alice_identity_project_001" + ], + "forbidden": [ + "bob_work_001" + ], + "resultIds": [ + "alice_oldest_memory_001", + "alice_identity_project_001", + "alice_anxiety_001", + "alice_pride_001", + "alice_last_week_work_001", + "alice_yesterday_work_001", + "alice_insight_001", + "alice_work_001", + "alice_high_importance_001", + "alice_work_003" + ], + "latencyMs": 45, + "pageSize": 10, + "top5Hits": [ + "alice_identity_project_001" + ], + "relevantAt10": [ + "alice_identity_project_001" + ], + "forbiddenHits": [], + "mrr": 0.5 + }, + { + "queryId": "cross_007", + "category": "cross_feature", + "expected": [ + "alice_high_importance_001" + ], + "forbidden": [], + "resultIds": [ + "alice_high_importance_001", + "alice_insight_001", + "alice_new_preference_001", + "alice_anxiety_001", + "alice_frustration_001", + "alice_work_001", + "alice_pride_001", + "alice_work_003", + "alice_work_002", + "alice_old_preference_001" + ], + "latencyMs": 37, + "pageSize": 10, + "top5Hits": [ + "alice_high_importance_001" + ], + "relevantAt10": [ + "alice_high_importance_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_008", + "category": "cross_feature", + "expected": [ + "alice_insight_001" + ], + "forbidden": [], + "resultIds": [ + "alice_insight_001", + "alice_emotion_change_001", + "alice_pride_001", + "alice_calm_001", + "alice_anxiety_001", + "alice_coffee_002", + "alice_yesterday_work_001", + "alice_stress_001", + "alice_coffee_001", + "alice_identity_project_001" + ], + "latencyMs": 41, + "pageSize": 10, + "top5Hits": [ + "alice_insight_001" + ], + "relevantAt10": [ + "alice_insight_001" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_009", + "category": "cross_feature", + "expected": [ + "eve_005" + ], + "forbidden": [], + "resultIds": [ + "eve_005", + "eve_007", + "eve_003", + "eve_009", + "eve_004", + "eve_001", + "eve_002", + "eve_008", + "eve_006", + "eve_010" + ], + "latencyMs": 31, + "pageSize": 10, + "top5Hits": [ + "eve_005" + ], + "relevantAt10": [ + "eve_005" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "cross_010", + "category": "cross_feature", + "expected": [ + "eve_007" + ], + "forbidden": [ + "alice_work_001" + ], + "resultIds": [ + "eve_007", + "eve_003", + "eve_005", + "eve_002", + "eve_001", + "eve_010", + "eve_004", + "eve_009", + "eve_006", + "eve_008" + ], + "latencyMs": 30, + "pageSize": 10, + "top5Hits": [ + "eve_007" + ], + "relevantAt10": [ + "eve_007" + ], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_006", + "category": "adversarial", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "alice_family_001", + "alice_work_001" + ], + "resultIds": [ + "bob_work_001", + "bob_family_002", + "bob_books_001", + "bob_coffee_002", + "bob_family_001", + "bob_health_001", + "bob_coffee_001", + "bob_work_002", + "bob_routine_001", + "bob_travel_001" + ], + "latencyMs": 23, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_007", + "category": "adversarial", + "expected": [], + "forbidden": [ + "bob_health_001" + ], + "resultIds": [ + "alice_calm_001", + "alice_joy_001", + "alice_worry_001", + "alice_health_002", + "alice_anxiety_001", + "alice_stress_001", + "alice_coffee_002", + "alice_work_002", + "alice_health_001", + "alice_family_003" + ], + "latencyMs": 38, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_008", + "category": "adversarial", + "expected": [], + "forbidden": [ + "alice_health_001", + "alice_health_002" + ], + "resultIds": [ + "bob_routine_001", + "bob_family_001", + "bob_health_001", + "bob_family_002", + "bob_coffee_002", + "bob_work_002", + "bob_work_001", + "bob_books_001", + "bob_coffee_001", + "bob_travel_001" + ], + "latencyMs": 34, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_009", + "category": "adversarial", + "expected": [], + "forbidden": [ + "carol_unicode_001", + "carol_short_001", + "carol_xss_001" + ], + "resultIds": [ + "eve_005", + "eve_007", + "eve_003", + "eve_001", + "eve_002", + "eve_006", + "eve_004", + "eve_009", + "eve_008", + "eve_010" + ], + "latencyMs": 22, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "adversarial_010", + "category": "adversarial", + "expected": [], + "forbidden": [ + "dave_today_001", + "dave_today_002" + ], + "resultIds": [ + "alice_joy_001", + "alice_pride_001", + "alice_work_002", + "alice_coffee_004_correction", + "alice_family_002", + "alice_grief_001", + "alice_calm_001", + "alice_emotion_change_001", + "alice_yesterday_work_001", + "alice_travel_001" + ], + "latencyMs": 28, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "edge_016", + "category": "edge_case", + "expected": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "forbidden": [ + "bob_coffee_001", + "bob_coffee_002" + ], + "resultIds": [ + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_coffee_002", + "alice_cooking_001", + "alice_travel_002", + "alice_worry_001", + "alice_travel_001", + "alice_books_001" + ], + "latencyMs": 38, + "pageSize": 10, + "top5Hits": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "relevantAt10": [ + "alice_coffee_001", + "alice_coffee_002" + ], + "forbiddenHits": [], + "mrr": 0.6 + }, + { + "queryId": "negative_001", + "category": "semantic", + "expected": [], + "forbidden": [ + "bob_coffee_001", + "carol_short_001", + "eve_001" + ], + "resultIds": [ + "alice_work_001", + "alice_identity_project_001", + "alice_new_preference_001", + "alice_yesterday_work_001", + "alice_coffee_003_old", + "alice_family_001", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_calm_001", + "alice_books_001" + ], + "latencyMs": 38, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "negative_002", + "category": "semantic", + "expected": [], + "forbidden": [ + "alice_coffee_001", + "carol_short_001" + ], + "resultIds": [ + "bob_family_001", + "bob_travel_001", + "bob_family_002", + "bob_books_001", + "bob_coffee_001", + "bob_health_001", + "bob_work_001", + "bob_work_002", + "bob_coffee_002", + "bob_routine_001" + ], + "latencyMs": 33, + "pageSize": 10, + "top5Hits": [], + "relevantAt10": [], + "forbiddenHits": [], + "mrr": 1 + }, + { + "queryId": "minimal_001", + "category": "semantic", + "expected": [ + "eve_009" + ], + "forbidden": [], + "resultIds": [ + "eve_003", + "eve_005", + "eve_007", + "eve_009", + "eve_002", + "eve_001", + "eve_004", + "eve_006", + "eve_008", + "eve_010" + ], + "latencyMs": 18, + "pageSize": 10, + "top5Hits": [ + "eve_009" + ], + "relevantAt10": [ + "eve_009" + ], + "forbiddenHits": [], + "mrr": 0.25 + } + ], + "repeatedResultIds": { + "semantic_001": [ + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_coffee_002", + "alice_cooking_001", + "alice_travel_002", + "alice_worry_001", + "alice_travel_001", + "alice_books_001" + ], + "semantic_002": [ + "alice_coffee_002", + "alice_calm_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_joy_001", + "alice_health_002", + "alice_coffee_001", + "alice_insight_001", + "alice_coffee_004_correction", + "alice_stress_001" + ], + "semantic_003": [ + "alice_work_001", + "alice_new_preference_001", + "alice_old_preference_001", + "alice_identity_project_001", + "alice_coffee_001", + "alice_yesterday_work_001", + "alice_frustration_001", + "alice_last_week_work_001", + "alice_work_002", + "alice_oldest_memory_001" + ], + "semantic_004": [ + "bob_coffee_001", + "bob_coffee_002", + "bob_routine_001", + "bob_travel_001", + "bob_health_001", + "bob_family_001", + "bob_books_001", + "bob_family_002", + "bob_work_002", + "bob_work_001" + ], + "semantic_005": [ + "alice_books_001", + "alice_health_002", + "alice_stress_001", + "alice_work_002", + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_travel_001", + "alice_joy_001", + "alice_emotion_change_001" + ], + "semantic_006": [ + "alice_cooking_001", + "alice_travel_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_family_001", + "alice_emotion_change_001", + "alice_joy_001", + "alice_old_preference_001", + "alice_work_001", + "alice_oldest_memory_001" + ], + "semantic_007": [ + "alice_finance_001", + "alice_worry_001", + "alice_health_002", + "alice_work_002", + "alice_calm_001", + "alice_pride_001", + "alice_insight_001", + "alice_joy_001", + "alice_cooking_001", + "alice_coffee_002" + ], + "semantic_008": [ + "bob_work_001", + "bob_coffee_002", + "bob_travel_001", + "bob_family_001", + "bob_health_001", + "bob_family_002", + "bob_books_001", + "bob_routine_001", + "bob_work_002", + "bob_coffee_001" + ], + "semantic_009": [ + "alice_travel_002", + "alice_travel_001", + "alice_work_003", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_calm_001", + "alice_coffee_004_correction", + "alice_family_002", + "alice_coffee_001", + "alice_work_002" + ], + "semantic_010": [ + "alice_work_003", + "alice_identity_project_001", + "alice_insight_001", + "alice_work_001", + "alice_work_002", + "alice_new_preference_001", + "alice_anxiety_001", + "alice_yesterday_work_001", + "alice_pride_001", + "alice_last_week_work_001" + ], + "semantic_011": [ + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_coffee_001", + "alice_coffee_002", + "alice_calm_001", + "alice_travel_002", + "alice_cooking_001", + "alice_worry_001", + "alice_emotion_change_001", + "alice_work_001" + ], + "emotional_001": [ + "alice_joy_001", + "alice_calm_001", + "alice_pride_001", + "alice_mixed_emotion_001", + "alice_family_001", + "alice_travel_001", + "alice_family_003", + "alice_low_importance_001", + "alice_emotion_change_001", + "alice_worry_001" + ], + "emotional_002": [ + "alice_stress_001", + "alice_grief_001", + "alice_anxiety_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_family_003", + "alice_work_002", + "alice_coffee_004_correction", + "alice_health_002", + "alice_calm_001" + ], + "emotional_003": [ + "alice_stress_001", + "alice_anxiety_001", + "alice_grief_001", + "alice_emotion_change_001", + "alice_family_003", + "alice_health_002", + "alice_work_002", + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_calm_001" + ], + "emotional_004": [ + "alice_worry_001", + "alice_anxiety_001", + "alice_stress_001", + "alice_grief_001", + "alice_work_002", + "alice_family_003", + "alice_health_002", + "alice_high_importance_001", + "alice_coffee_001", + "alice_mixed_emotion_001" + ], + "emotional_005": [ + "alice_frustration_001", + "alice_stress_001", + "alice_anxiety_001", + "alice_emotion_change_001", + "alice_worry_001", + "alice_work_002", + "alice_coffee_004_correction", + "alice_family_003", + "alice_new_preference_001", + "alice_coffee_002" + ], + "emotional_006": [ + "alice_pride_001", + "alice_joy_001", + "alice_travel_001", + "alice_health_002", + "alice_books_001", + "alice_family_003", + "alice_oldest_memory_001", + "alice_work_002", + "alice_emotion_change_001", + "alice_grief_001" + ], + "emotional_007": [ + "alice_anxiety_001", + "alice_stress_001", + "alice_worry_001", + "alice_emotion_change_001", + "alice_grief_001", + "alice_work_002", + "alice_coffee_002", + "alice_family_003", + "alice_coffee_004_correction", + "alice_calm_001" + ], + "emotional_008": [ + "alice_mixed_emotion_001", + "alice_joy_001", + "alice_calm_001", + "alice_pride_001", + "alice_finance_001", + "alice_work_002", + "alice_family_003", + "alice_coffee_002", + "alice_worry_001", + "alice_emotion_change_001" + ], + "emotional_009": [ + "alice_emotion_change_001", + "alice_pride_001", + "alice_calm_001", + "alice_health_002", + "alice_coffee_001", + "alice_insight_001", + "alice_worry_001", + "alice_work_002", + "alice_anxiety_001", + "alice_family_003" + ], + "emotional_010": [ + "alice_calm_001", + "alice_anxiety_001", + "alice_joy_001", + "alice_health_002", + "alice_stress_001", + "alice_family_003", + "alice_travel_001", + "alice_emotion_change_001", + "alice_grief_001", + "alice_mixed_emotion_001" + ], + "temporal_001": [ + "dave_today_001", + "dave_today_002" + ], + "temporal_002": [ + "dave_today_001", + "dave_today_002" + ], + "temporal_003": [ + "alice_family_001", + "alice_stress_001", + "alice_family_003", + "alice_work_002", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_recent_convo_001" + ], + "temporal_004": [ + "alice_health_002", + "alice_joy_001", + "alice_work_002", + "alice_stress_001", + "alice_yesterday_work_001", + "alice_last_week_work_001", + "alice_recent_convo_001", + "alice_calm_001", + "alice_worry_001", + "alice_family_003" + ], + "temporal_005": [ + "alice_grief_001", + "alice_travel_001", + "alice_stress_001", + "alice_family_002", + "alice_identity_project_001", + "alice_pride_001", + "alice_oldest_memory_001", + "alice_joy_001", + "alice_family_001", + "alice_cooking_001" + ], + "temporal_006": [ + "alice_work_002", + "alice_recent_convo_001", + "alice_stress_001", + "alice_family_003", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_family_001" + ], + "temporal_007": [ + "alice_yesterday_work_001", + "alice_stress_001", + "alice_work_002", + "alice_recent_convo_001", + "alice_frustration_001", + "alice_family_003", + "alice_family_001" + ], + "temporal_008": [ + "alice_new_preference_001", + "alice_old_preference_001", + "alice_oldest_memory_001", + "alice_work_001", + "alice_insight_001", + "alice_last_week_work_001", + "alice_frustration_001", + "alice_yesterday_work_001", + "alice_identity_project_001", + "alice_work_002" + ], + "temporal_009": [], + "temporal_010": [], + "temporal_011": [ + "alice_oldest_memory_001", + "alice_new_preference_001", + "alice_old_preference_001", + "alice_pride_001", + "alice_insight_001", + "alice_identity_project_001", + "alice_health_002", + "alice_work_001", + "alice_stress_001", + "alice_work_002" + ], + "rls_001": [ + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_003_old", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_low_importance_001", + "alice_travel_001" + ], + "rls_002": [ + "bob_coffee_001", + "bob_routine_001", + "bob_coffee_002", + "bob_travel_001", + "bob_health_001", + "bob_family_001", + "bob_family_002", + "bob_work_002", + "bob_work_001", + "bob_books_001" + ], + "rls_003": [ + "alice_family_001", + "alice_cooking_001", + "alice_joy_001", + "alice_family_003", + "alice_finance_001", + "alice_travel_001", + "alice_family_002", + "alice_health_002", + "alice_worry_001", + "alice_grief_001" + ], + "rls_004": [ + "bob_family_001", + "bob_family_002", + "bob_work_002", + "bob_coffee_002", + "bob_travel_001", + "bob_health_001", + "bob_routine_001", + "bob_work_001", + "bob_coffee_001", + "bob_books_001" + ], + "rls_005": [ + "alice_travel_001", + "alice_cooking_001", + "alice_travel_002", + "alice_coffee_004_correction", + "alice_family_002", + "alice_joy_001", + "alice_calm_001", + "alice_coffee_003_old", + "alice_family_001", + "alice_coffee_001" + ], + "rls_006": [ + "bob_travel_001", + "bob_coffee_002", + "bob_health_001", + "bob_coffee_001", + "bob_work_002", + "bob_family_002", + "bob_books_001", + "bob_family_001", + "bob_work_001", + "bob_routine_001" + ], + "rls_007": [ + "alice_work_001", + "alice_insight_001", + "alice_new_preference_001", + "alice_oldest_memory_001", + "alice_pride_001", + "alice_work_002", + "alice_old_preference_001", + "alice_identity_project_001", + "alice_anxiety_001", + "alice_yesterday_work_001" + ], + "rls_008": [ + "alice_health_001", + "alice_calm_001", + "alice_joy_001", + "alice_new_preference_001", + "alice_health_002", + "alice_cooking_001", + "alice_travel_002", + "alice_anxiety_001", + "alice_travel_001", + "alice_work_002" + ], + "rls_009": [ + "bob_health_001", + "bob_family_001", + "bob_travel_001", + "bob_family_002", + "bob_routine_001", + "bob_books_001", + "bob_coffee_002", + "bob_work_002", + "bob_coffee_001", + "bob_work_001" + ], + "rls_010": [ + "bob_routine_001", + "bob_family_001", + "bob_coffee_002", + "bob_coffee_001", + "bob_health_001", + "bob_work_002", + "bob_family_002", + "bob_travel_001", + "bob_work_001", + "bob_books_001" + ], + "adversarial_001": [ + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_cooking_001", + "alice_travel_002", + "alice_travel_001", + "alice_emotion_change_001", + "alice_work_001" + ], + "adversarial_002": [ + "alice_high_importance_001", + "alice_health_001", + "alice_pride_001", + "alice_identity_project_001", + "alice_family_001", + "alice_work_003", + "alice_finance_001", + "alice_grief_001", + "alice_address_001", + "alice_phone_001" + ], + "adversarial_003": [ + "alice_family_001", + "alice_family_002", + "alice_mixed_emotion_001", + "alice_worry_001", + "alice_coffee_004_correction", + "alice_recent_convo_001", + "alice_work_002", + "alice_identity_project_001", + "alice_family_003", + "alice_travel_002" + ], + "adversarial_004": [ + "alice_mixed_emotion_001", + "alice_family_003", + "alice_grief_001", + "alice_cooking_001", + "alice_joy_001", + "alice_pride_001", + "alice_finance_001", + "alice_travel_001", + "alice_family_001", + "alice_coffee_004_correction" + ], + "adversarial_005": [ + "bob_travel_001", + "bob_coffee_002", + "bob_family_002", + "bob_health_001", + "bob_family_001", + "bob_coffee_001", + "bob_books_001", + "bob_routine_001", + "bob_work_002", + "bob_work_001" + ], + "edge_001": [ + "eve_003", + "eve_005", + "eve_007", + "eve_009", + "eve_006", + "eve_008", + "eve_004", + "eve_010", + "eve_002", + "eve_001" + ], + "edge_002": [ + "eve_003", + "eve_005", + "eve_007", + "eve_006", + "eve_004", + "eve_002", + "eve_009", + "eve_010", + "eve_008", + "eve_001" + ], + "edge_003": [ + "carol_xss_001", + "carol_xss_003", + "carol_xss_004", + "carol_xss_002", + "carol_short_001", + "carol_sqli_002", + "carol_null_003", + "carol_unicode_002", + "carol_unicode_004", + "carol_special_003" + ], + "edge_004": [ + "carol_sqli_001", + "carol_long_001", + "carol_unicode_002", + "carol_unicode_001", + "carol_short_003", + "carol_unicode_005", + "carol_null_003", + "carol_short_001", + "carol_short_002", + "carol_unicode_004" + ], + "edge_005": [ + "carol_unicode_001", + "carol_unicode_004", + "carol_short_001", + "carol_short_003", + "carol_unicode_002", + "carol_unicode_003", + "carol_null_003", + "carol_short_002", + "carol_xss_002", + "carol_unicode_005" + ], + "edge_006": [], + "edge_007": [ + "alice_pride_001", + "alice_identity_project_001", + "alice_new_preference_001", + "alice_work_001", + "alice_books_001", + "alice_old_preference_001", + "alice_stress_001", + "alice_travel_001", + "alice_work_003", + "alice_work_002" + ], + "edge_008": [ + "carol_unicode_002", + "carol_null_003", + "carol_short_001", + "carol_unicode_003", + "carol_short_003", + "carol_unicode_004", + "carol_xss_002", + "carol_unicode_001", + "carol_xss_001", + "carol_short_002" + ], + "edge_009": [ + "carol_sqli_003", + "carol_sqli_002", + "carol_short_001", + "carol_short_003", + "carol_null_003", + "carol_xss_002", + "carol_xss_001", + "carol_special_001", + "carol_short_002", + "carol_unicode_003" + ], + "edge_010": [ + "alice_identity_project_001", + "alice_yesterday_work_001", + "alice_books_001", + "alice_recent_convo_001", + "alice_work_003", + "alice_last_week_work_001", + "alice_anxiety_001", + "alice_new_preference_001", + "alice_family_002", + "alice_coffee_003_old" + ], + "edge_011": [ + "alice_pride_001", + "alice_joy_001", + "alice_work_002", + "alice_family_002", + "alice_anxiety_001", + "alice_family_003", + "alice_worry_001", + "alice_finance_001", + "alice_calm_001", + "alice_new_preference_001" + ], + "edge_012": [ + "alice_calm_001", + "alice_coffee_001", + "alice_coffee_002", + "alice_coffee_003_old", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_emotion_change_001", + "alice_cooking_001", + "alice_low_importance_001", + "alice_travel_001" + ], + "edge_013": [ + "alice_phone_001", + "alice_address_001", + "alice_worry_001", + "alice_grief_001", + "alice_work_002", + "alice_anxiety_001", + "alice_family_002", + "alice_joy_001", + "alice_stress_001", + "alice_health_002" + ], + "edge_014": [ + "alice_address_001", + "alice_phone_001", + "alice_joy_001", + "alice_pride_001", + "alice_grief_001", + "alice_worry_001", + "alice_work_002", + "alice_identity_project_001", + "alice_emotion_change_001", + "alice_family_002" + ], + "edge_015": [ + "eve_005", + "eve_007", + "eve_003", + "eve_004", + "eve_006", + "eve_008", + "eve_002", + "eve_009", + "eve_010", + "eve_001" + ], + "cross_001": [ + "alice_health_001", + "alice_calm_001", + "alice_coffee_002", + "alice_emotion_change_001", + "alice_stress_001", + "alice_coffee_001", + "alice_family_003", + "alice_coffee_003_old", + "alice_cooking_001", + "alice_joy_001" + ], + "cross_002": [ + "alice_health_002", + "alice_calm_001", + "alice_joy_001", + "alice_cooking_001", + "alice_insight_001", + "alice_family_003", + "alice_pride_001", + "alice_finance_001", + "alice_high_importance_001", + "alice_travel_001" + ], + "cross_003": [ + "alice_finance_001", + "alice_worry_001", + "alice_mixed_emotion_001", + "alice_joy_001", + "alice_work_002", + "alice_calm_001", + "alice_health_002", + "alice_anxiety_001", + "alice_travel_001", + "alice_family_002" + ], + "cross_004": [ + "alice_family_003", + "alice_joy_001", + "alice_cooking_001", + "alice_family_001", + "alice_mixed_emotion_001", + "alice_calm_001", + "alice_work_002", + "alice_health_002", + "alice_finance_001", + "alice_worry_001" + ], + "cross_005": [ + "bob_family_001", + "bob_routine_001", + "bob_coffee_002", + "bob_family_002", + "bob_work_002", + "bob_travel_001", + "bob_coffee_001", + "bob_work_001", + "bob_health_001", + "bob_books_001" + ], + "cross_006": [ + "alice_oldest_memory_001", + "alice_identity_project_001", + "alice_anxiety_001", + "alice_pride_001", + "alice_last_week_work_001", + "alice_yesterday_work_001", + "alice_insight_001", + "alice_work_001", + "alice_high_importance_001", + "alice_work_003" + ], + "cross_007": [ + "alice_high_importance_001", + "alice_insight_001", + "alice_new_preference_001", + "alice_anxiety_001", + "alice_frustration_001", + "alice_work_001", + "alice_pride_001", + "alice_work_003", + "alice_work_002", + "alice_old_preference_001" + ], + "cross_008": [ + "alice_insight_001", + "alice_emotion_change_001", + "alice_pride_001", + "alice_calm_001", + "alice_anxiety_001", + "alice_coffee_002", + "alice_yesterday_work_001", + "alice_stress_001", + "alice_coffee_001", + "alice_identity_project_001" + ], + "cross_009": [ + "eve_005", + "eve_007", + "eve_003", + "eve_009", + "eve_004", + "eve_001", + "eve_002", + "eve_008", + "eve_006", + "eve_010" + ], + "cross_010": [ + "eve_007", + "eve_003", + "eve_005", + "eve_002", + "eve_001", + "eve_010", + "eve_004", + "eve_009", + "eve_006", + "eve_008" + ], + "adversarial_006": [ + "bob_work_001", + "bob_family_002", + "bob_books_001", + "bob_coffee_002", + "bob_family_001", + "bob_health_001", + "bob_coffee_001", + "bob_work_002", + "bob_routine_001", + "bob_travel_001" + ], + "adversarial_007": [ + "alice_calm_001", + "alice_joy_001", + "alice_worry_001", + "alice_health_002", + "alice_anxiety_001", + "alice_stress_001", + "alice_coffee_002", + "alice_work_002", + "alice_health_001", + "alice_family_003" + ], + "adversarial_008": [ + "bob_routine_001", + "bob_family_001", + "bob_health_001", + "bob_family_002", + "bob_coffee_002", + "bob_work_002", + "bob_work_001", + "bob_books_001", + "bob_coffee_001", + "bob_travel_001" + ], + "adversarial_009": [ + "eve_005", + "eve_007", + "eve_003", + "eve_001", + "eve_002", + "eve_006", + "eve_004", + "eve_009", + "eve_008", + "eve_010" + ], + "adversarial_010": [ + "alice_joy_001", + "alice_pride_001", + "alice_work_002", + "alice_coffee_004_correction", + "alice_family_002", + "alice_grief_001", + "alice_calm_001", + "alice_emotion_change_001", + "alice_yesterday_work_001", + "alice_travel_001" + ], + "edge_016": [ + "alice_coffee_001", + "alice_coffee_004_correction", + "alice_coffee_003_old", + "alice_calm_001", + "alice_coffee_002", + "alice_cooking_001", + "alice_travel_002", + "alice_worry_001", + "alice_travel_001", + "alice_books_001" + ], + "negative_001": [ + "alice_work_001", + "alice_identity_project_001", + "alice_new_preference_001", + "alice_yesterday_work_001", + "alice_coffee_003_old", + "alice_family_001", + "alice_coffee_004_correction", + "alice_joy_001", + "alice_calm_001", + "alice_books_001" + ], + "negative_002": [ + "bob_family_001", + "bob_travel_001", + "bob_family_002", + "bob_books_001", + "bob_coffee_001", + "bob_health_001", + "bob_work_001", + "bob_work_002", + "bob_coffee_002", + "bob_routine_001" + ], + "minimal_001": [ + "eve_003", + "eve_005", + "eve_007", + "eve_009", + "eve_002", + "eve_001", + "eve_004", + "eve_006", + "eve_008", + "eve_010" + ] + } + }, + "comparison": { + "top5Gains": [], + "top5Losses": [], + "rankingChanges": [ + "emotional_001", + "emotional_002", + "emotional_003", + "emotional_004", + "emotional_005", + "emotional_006", + "emotional_007", + "emotional_008" + ] + } +} \ No newline at end of file diff --git a/scripts/research/retrieval-probe.mjs b/scripts/research/retrieval-probe.mjs index 91605cd..6dcd84c 100644 --- a/scripts/research/retrieval-probe.mjs +++ b/scripts/research/retrieval-probe.mjs @@ -20,14 +20,22 @@ */ import { readFileSync, readdirSync, writeFileSync } from 'node:fs'; +import { execFileSync } from 'node:child_process'; import { join } from 'node:path'; const MNEMON = process.env.PROBE_MNEMON ?? '/Users/beauxwalton/projects/mnemon'; -const BASE = (process.env.PROBE_BASE_URL ?? 'http://127.0.0.1:3007').replace(/\/+$/, ''); +const BASE = (process.env.PROBE_BASE_URL ?? 'http://127.0.0.1:3007').replace( + /\/+$/, + '', +); const KEY = process.env.PROBE_API_KEY; const USER = process.env.PROBE_USER_ID ?? 'mnemon-v02-noise-prefix'; -const TASKS = process.env.PROBE_TASKS_DIR ?? join(MNEMON, '.mnemon/noisy-tasks-prefix'); -const RECEIPT = process.env.PROBE_RECEIPT ?? join(MNEMON, '.mnemon/noisy-corpus-receipt-prefix.json'); +const TASKS = + process.env.PROBE_TASKS_DIR ?? join(MNEMON, '.mnemon/noisy-tasks-prefix'); +const RECEIPT = + process.env.PROBE_RECEIPT ?? + join(MNEMON, '.mnemon/noisy-corpus-receipt-prefix.json'); +const RESET_DB = process.env.PROBE_RESET_DB_URL; if (!KEY) { process.stderr.write('PROBE_API_KEY is required\n'); @@ -46,7 +54,9 @@ const EXTRA_HEADERS = {}; const flagHeader = arg('header', null); // e.g. --header 'X-Engram-Rescue-Mode: relative' if (flagHeader) { const idx = flagHeader.indexOf(':'); - EXTRA_HEADERS[flagHeader.slice(0, idx).trim()] = flagHeader.slice(idx + 1).trim(); + EXTRA_HEADERS[flagHeader.slice(0, idx).trim()] = flagHeader + .slice(idx + 1) + .trim(); } const receipt = JSON.parse(readFileSync(RECEIPT, 'utf8')); @@ -79,7 +89,31 @@ async function recall(query, limit) { const text = await res.text(); if (!res.ok) throw new Error(`recall ${res.status}: ${text.slice(0, 400)}`); const data = JSON.parse(text); - return data.memories ?? []; + return { + memories: data.memories ?? [], + latencyMs: Number(data.latencyMs ?? 0), + }; +} + +function resetUsage() { + if (!RESET_DB) return; + execFileSync( + 'psql', + [ + RESET_DB, + '-q', + '-c', + `UPDATE memories m + SET retrieval_count = s.retrieval_count, + used_count = s.used_count, + unused_count = s.unused_count, + last_retrieved_at = s.last_retrieved_at, + last_used_at = s.last_used_at + FROM zz_probe_usage_snapshot s + WHERE m.id = s.id;`, + ], + { stdio: 'ignore' }, + ); } const rows = []; @@ -97,9 +131,13 @@ for (const file of taskFiles) { } // Ranked page the benchmark would actually consume. - const page = await recall(task.query, LIMIT); + resetUsage(); + const pageResponse = await recall(task.query, LIMIT); + const page = pageResponse.memories; // Deep page: is the gold anywhere in the candidate pool at all? - const deep = await recall(task.query, 250); + resetUsage(); + const deepResponse = await recall(task.query, 250); + const deep = deepResponse.memories; const rank = page.findIndex((m) => m.id === gold); // -1 = absent const deepRank = deep.findIndex((m) => m.id === gold); @@ -116,7 +154,10 @@ for (const file of taskFiles) { goldId: gold, rankInPage: rank, rankInPool: deepRank, + pageSize: page.length, poolSize: deep.length, + pageLatencyMs: pageResponse.latencyMs, + deepLatencyMs: deepResponse.latencyMs, goldScore: goldRow ? goldRow.score : null, goldBand: goldRow ? band(goldRow.score) : 'ABSENT', top: page.slice(0, 10).map((m, i) => ({ @@ -165,7 +206,9 @@ for (const r of rows) { ); } } -process.stdout.write(`\n--- summary ---\n${JSON.stringify(summary, null, 2)}\n`); +process.stdout.write( + `\n--- summary ---\n${JSON.stringify(summary, null, 2)}\n`, +); if (OUT) { writeFileSync(OUT, JSON.stringify({ summary, rows }, null, 2)); diff --git a/scripts/research/run-candidate-matrix.sh b/scripts/research/run-candidate-matrix.sh new file mode 100755 index 0000000..4e46a72 --- /dev/null +++ b/scripts/research/run-candidate-matrix.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +set -euo pipefail + +REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +DB="${PROBE_DB_URL:?Set PROBE_DB_URL to a dedicated local engram_ablation_* database}" +RUN_PORT="${PROBE_PORT:-3017}" +KEY="${PROBE_API_KEY:?Set PROBE_API_KEY for the isolated research fixture}" +OUT_DIR="${CANDIDATE_OUT_DIR:-$REPO/docs/research/memory-formation-query-transform/artifacts/candidate-matrix/isolated}" +mkdir -p "$OUT_DIR" + +DB_HOST="$(node -e 'process.stdout.write(new URL(process.argv[1]).hostname)' "$DB")" +DB_NAME="$(psql "$DB" -Atqc 'SELECT current_database()')" +if [[ "$DB_HOST" != "localhost" && "$DB_HOST" != "127.0.0.1" && "$DB_HOST" != "::1" ]]; then + echo "candidate matrix refuses non-local database host: $DB_HOST" >&2 + exit 1 +fi +if [[ "$DB_NAME" != engram_ablation_* ]]; then + echo "candidate matrix requires a dedicated engram_ablation_* database; got $DB_NAME" >&2 + exit 1 +fi +if ! psql "$DB" -Atqc "SELECT to_regclass('zz_probe_usage_snapshot') IS NOT NULL" | grep -qx t; then + echo "required usage snapshot zz_probe_usage_snapshot is missing" >&2 + exit 1 +fi + +reset_usage() { + psql "$DB" -q -c " + UPDATE memories m + SET retrieval_count = s.retrieval_count, + used_count = s.used_count, + unused_count = s.unused_count, + last_retrieved_at = s.last_retrieved_at, + last_used_at = s.last_used_at + FROM zz_probe_usage_snapshot s + WHERE m.id = s.id;" >/dev/null +} + +stop_server() { + if [[ -n "${SERVER_PID:-}" ]]; then + kill "$SERVER_PID" 2>/dev/null || true + wait "$SERVER_PID" 2>/dev/null || true + SERVER_PID="" + fi +} +trap stop_server EXIT + +start_server() { + local label="$1" depth="$2" cluster_limit="$3" + stop_server + ( + cd "$REPO" + DATABASE_URL="$DB" PORT="$RUN_PORT" \ + RECALL_RERANK_SCALE_FIX=true \ + RECALL_LEXICAL_COVERAGE_FLOOR=true \ + RECALL_RESCUE_SQL_TIEBREAK=true \ + RECALL_CANDIDATE_POOL_DEPTH="$depth" \ + RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT="$cluster_limit" \ + exec node dist/main.js + ) >"$OUT_DIR/${label}.server.log" 2>&1 & + SERVER_PID=$! + for _ in $(seq 1 60); do + curl -sf -m 2 "http://127.0.0.1:${RUN_PORT}/v1/health" >/dev/null 2>&1 && return 0 + sleep 1 + done + echo "server did not start for $label" >&2 + exit 1 +} + +# logical arm : candidate depth : max selected per >=0.9 Jaccard cluster +ARMS=( + "depth10:10:" + "depth12:12:" + "depth20:20:" + "depth50:50:" + "cluster12:12:2" + "cluster20:20:2" + "cluster50:50:2" + "cluster12c8:12:8" + "cluster20c4:20:4" + "cluster20c8:20:8" + "cluster50c4:50:4" + "cluster50c8:50:8" +) +if [[ -n "${CANDIDATE_ARMS:-}" ]]; then + IFS=',' read -r -a ARMS <<<"$CANDIDATE_ARMS" +fi + +ARTIFACTS=() +for spec in "${ARMS[@]}"; do + IFS=: read -r arm depth cluster_limit <<<"$spec" + for repeat in 1 2; do + label="${arm}-r${repeat}" + echo "### $label depth=$depth cluster_limit=${cluster_limit:-off}" + start_server "$label" "$depth" "$cluster_limit" + reset_usage + PROBE_API_KEY="$KEY" \ + PROBE_BASE_URL="http://127.0.0.1:${RUN_PORT}" \ + PROBE_RESET_DB_URL="$DB" \ + node "$REPO/scripts/research/retrieval-probe.mjs" \ + --limit 10 --label "$label" --out "$OUT_DIR/${label}.json" \ + | tail -4 + ARTIFACTS+=("$OUT_DIR/${label}.json") + done +done + +node "$REPO/scripts/research/score-candidate-matrix.mjs" "${ARTIFACTS[@]}" \ + | tee "$OUT_DIR/summary.txt" diff --git a/scripts/research/run-diversity-production-gate.ts b/scripts/research/run-diversity-production-gate.ts new file mode 100644 index 0000000..6daec89 --- /dev/null +++ b/scripts/research/run-diversity-production-gate.ts @@ -0,0 +1,401 @@ +#!/usr/bin/env ts-node +import 'dotenv/config'; +import { writeFileSync } from 'node:fs'; +import request from 'supertest'; +import { createTestApp } from '../../test/helpers/create-test-app.js'; +import { seedCorpus, type SeededUser } from '../../test/helpers/seed-corpus.js'; +import { generateCorpusEmbeddings } from '../../test/helpers/generate-embeddings.js'; +import { asUser } from '../../test/helpers/auth-helpers.js'; +import { ALL_USERS, GOLD_QUERIES } from '../../test/fixtures/index.js'; +import type { GoldQuery } from '../../test/fixtures/types.js'; +import { EmbeddingService as EmbeddingGeneratorService } from '../../src/embedding/embedding.service.js'; + +const OUT = process.env.DIVERSITY_GATE_OUT; +if (!OUT) throw new Error('DIVERSITY_GATE_OUT is required'); +const OUTPUT_PATH: string = OUT; + +const FINAL_LIMIT = 10; +const SNAPSHOT = 'zz_diversity_gate_usage_snapshot_20260823'; + +type AppContext = Awaited>; +type SeededCorpus = Awaited>; + +interface QueryRun { + queryId: string; + category: string; + expected: string[]; + forbidden: string[]; + resultIds: string[]; + latencyMs: number; + pageSize: number; + top5Hits: string[]; + relevantAt10: string[]; + forbiddenHits: string[]; + mrr: number; +} + +interface ArmResult { + label: string; + summary: ReturnType; + stabilityChangedQueries: string[]; + queries: QueryRun[]; + repeatedResultIds: Record; +} + +function configureArm(diversity: boolean): void { + process.env.RECALL_RERANK_SCALE_FIX = 'true'; + process.env.RECALL_LEXICAL_COVERAGE_FLOOR = 'true'; + process.env.RECALL_RESCUE_SQL_TIEBREAK = 'true'; + if (diversity) { + process.env.RECALL_CANDIDATE_POOL_DEPTH = '12'; + process.env.RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT = '2'; + } else { + delete process.env.RECALL_CANDIDATE_POOL_DEPTH; + delete process.env.RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT; + } +} + +async function resetUsage(ctx: AppContext): Promise { + await ctx.prisma.$executeRawUnsafe(` + UPDATE memories m + SET retrieval_count = s.retrieval_count, + used_count = s.used_count, + unused_count = s.unused_count, + last_retrieved_at = s.last_retrieved_at, + last_used_at = s.last_used_at + FROM ${SNAPSHOT} s + WHERE m.id = s.id + `); +} + +async function assertIsolatedLocalDatabase(ctx: AppContext): Promise { + const databaseUrl = process.env.DATABASE_URL; + if (!databaseUrl) throw new Error('DATABASE_URL is required'); + const parsed = new URL(databaseUrl); + if (!['localhost', '127.0.0.1', '::1'].includes(parsed.hostname)) { + throw new Error('diversity gate refuses non-local DATABASE_URL'); + } + const rows = await ctx.prisma.$queryRawUnsafe>( + 'SELECT current_database() AS database', + ); + const database = rows[0]?.database ?? ''; + if (!/^engram_diversity_gate_[a-z0-9_]+$/i.test(database)) { + throw new Error( + `diversity gate requires a dedicated engram_diversity_gate_* database; got ${database}`, + ); + } +} + +async function restoreDeclaredFixtureFields( + ctx: AppContext, + corpus: Awaited>, +): Promise<{ memoriesChecked: number; taggedMemories: number }> { + const expected = new Map( + ALL_USERS.flatMap((user) => + user.memories.map((memory) => [memory.fixture_id, memory] as const), + ), + ); + + // Issue #326: seedCorpus currently omits tags, metadata, and memoryType. + // Restore every declared retrieval field before this gate so the benchmark + // measures production behavior rather than a silently degraded fixture. + for (const memory of expected.values()) { + await ctx.prisma.$executeRawUnsafe( + `UPDATE memories + SET tags = $1::text[], + metadata = $2::jsonb, + memory_type = $3::"MemoryType" + WHERE id = $4`, + memory.tags, + JSON.stringify(memory.metadata ?? null), + memory.memoryType ?? null, + memory.fixture_id, + ); + } + + const rows = await ctx.prisma.memory.findMany({ + where: { userId: { in: corpus.seededUsers.map((user) => user.userId) } }, + select: { id: true, tags: true, metadata: true, memoryType: true }, + }); + if (rows.length !== expected.size) { + throw new Error( + `fixture integrity: expected ${expected.size} memories, found ${rows.length}`, + ); + } + for (const row of rows) { + const fixture = expected.get(row.id); + if (!fixture) throw new Error(`unexpected fixture memory ${row.id}`); + if (JSON.stringify(row.tags) !== JSON.stringify(fixture.tags)) { + throw new Error(`fixture tag mismatch for ${row.id}`); + } + if ((row.memoryType ?? null) !== (fixture.memoryType ?? null)) { + throw new Error(`fixture memoryType mismatch for ${row.id}`); + } + if ( + JSON.stringify(row.metadata ?? null) !== + JSON.stringify(fixture.metadata ?? null) + ) { + throw new Error(`fixture metadata mismatch for ${row.id}`); + } + } + return { + memoriesChecked: rows.length, + taggedMemories: rows.filter((row) => row.tags.length > 0).length, + }; +} + +async function execute( + ctx: AppContext, + users: Map, + query: GoldQuery, +): Promise { + if (!query.query.trim()) { + return score(query, [], 0); + } + const user = users.get(query.user); + if (!user) throw new Error(`unknown fixture user: ${query.user}`); + await resetUsage(ctx); + const response = await request(ctx.app.getHttpServer()) + .post('/v1/memories/query') + .set(asUser(user.apiKey, user.userId)) + .send({ query: query.query, limit: FINAL_LIMIT }); + if (![200, 201, 400].includes(response.status)) { + throw new Error( + `${query.id}: status=${response.status} ${JSON.stringify(response.body)}`, + ); + } + const memories = + response.status === 400 ? [] : (response.body.memories ?? []); + return score( + query, + memories.map((memory: { id: string }) => memory.id), + Number(response.body.latencyMs ?? 0), + ); +} + +function score(query: GoldQuery, ids: string[], latencyMs: number): QueryRun { + const expected = [...query.must_top5, ...(query.should_top20 ?? [])]; + const top5Hits = query.must_top5.filter((id) => ids.slice(0, 5).includes(id)); + const relevantAt10 = expected.filter((id) => ids.includes(id)); + const forbiddenHits = query.must_absent.filter((id) => ids.includes(id)); + const reciprocalRanks = query.must_top5.map((id) => { + const index = ids.indexOf(id); + return index >= 0 ? 1 / (index + 1) : 0; + }); + return { + queryId: query.id, + category: query.category, + expected, + forbidden: query.must_absent, + resultIds: ids, + latencyMs, + pageSize: ids.length, + top5Hits, + relevantAt10, + forbiddenHits, + mrr: + reciprocalRanks.length > 0 + ? reciprocalRanks.reduce((sum, value) => sum + value, 0) / + reciprocalRanks.length + : 1, + }; +} + +function percentile(values: number[], p: number): number { + if (values.length === 0) return 0; + const sorted = [...values].sort((a, b) => a - b); + return sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * p))]; +} + +function summarize(queries: QueryRun[]) { + const mustTop5Count = GOLD_QUERIES.reduce( + (sum, query) => sum + query.must_top5.length, + 0, + ); + const top5HitCount = queries.reduce( + (sum, query) => sum + query.top5Hits.length, + 0, + ); + const expectedCount = queries.reduce( + (sum, query) => sum + query.expected.length, + 0, + ); + const relevantAt10Count = queries.reduce( + (sum, query) => sum + query.relevantAt10.length, + 0, + ); + const forbiddenCount = queries.reduce( + (sum, query) => sum + query.forbidden.length, + 0, + ); + const forbiddenHitCount = queries.reduce( + (sum, query) => sum + query.forbiddenHits.length, + 0, + ); + const latencies = queries + .filter((query) => query.latencyMs > 0) + .map((query) => query.latencyMs); + const nonEmpty = queries.filter((query) => query.queryId !== 'edge_006'); + return { + queryCount: queries.length, + precisionAt5: mustTop5Count > 0 ? top5HitCount / mustTop5Count : 1, + recallAt10: expectedCount > 0 ? relevantAt10Count / expectedCount : 1, + mrrAt10: + queries.reduce((sum, query) => sum + query.mrr, 0) / queries.length, + falsePositiveInjectionRate: + forbiddenCount > 0 ? forbiddenHitCount / forbiddenCount : 0, + queriesWithFalsePositiveInjection: queries.filter( + (query) => query.forbiddenHits.length > 0, + ).length, + fullPageRate: + nonEmpty.filter((query) => query.pageSize === FINAL_LIMIT).length / + nonEmpty.length, + minPageSize: Math.min(...nonEmpty.map((query) => query.pageSize)), + meanPageSize: + nonEmpty.reduce((sum, query) => sum + query.pageSize, 0) / + nonEmpty.length, + latencyP50Ms: percentile(latencies, 0.5), + latencyP95Ms: percentile(latencies, 0.95), + latencyMeanMs: + latencies.reduce((sum, value) => sum + value, 0) / latencies.length, + _counts: { + mustTop5Count, + top5HitCount, + expectedCount, + relevantAt10Count, + forbiddenCount, + forbiddenHitCount, + }, + }; +} + +async function runArm( + label: string, + ctx: AppContext, + users: Map, +): Promise { + const first: QueryRun[] = []; + const repeatedResultIds: Record = {}; + const stabilityChangedQueries: string[] = []; + for (const query of GOLD_QUERIES) { + const primary = await execute(ctx, users, query); + const repeat = await execute(ctx, users, query); + first.push(primary); + repeatedResultIds[query.id] = repeat.resultIds; + if (primary.resultIds.join(',') !== repeat.resultIds.join(',')) { + stabilityChangedQueries.push(query.id); + } + process.stdout.write('.'); + } + process.stdout.write(` ${label}\n`); + return { + label, + summary: summarize(first), + stabilityChangedQueries, + queries: first, + repeatedResultIds, + }; +} + +function compare(off: ArmResult, on: ArmResult) { + const offById = new Map(off.queries.map((query) => [query.queryId, query])); + const gains: string[] = []; + const losses: string[] = []; + const rankingChanges: string[] = []; + for (const query of on.queries) { + const before = offById.get(query.queryId)!; + if (query.top5Hits.length > before.top5Hits.length) + gains.push(query.queryId); + if (query.top5Hits.length < before.top5Hits.length) + losses.push(query.queryId); + if (query.resultIds.join(',') !== before.resultIds.join(',')) { + rankingChanges.push(query.queryId); + } + } + return { top5Gains: gains, top5Losses: losses, rankingChanges }; +} + +async function main(): Promise { + let offContext: AppContext | undefined; + let onContext: AppContext | undefined; + let corpus: SeededCorpus | undefined; + let snapshotCreated = false; + try { + configureArm(false); + offContext = await createTestApp(false); + await assertIsolatedLocalDatabase(offContext); + corpus = await seedCorpus(offContext.prisma); + const fixtureIntegrity = await restoreDeclaredFixtureFields( + offContext, + corpus, + ); + await offContext.prisma.account.updateMany({ + where: { id: { in: corpus.seededUsers.map((user) => user.accountId) } }, + data: { plan: 'SCALE' }, + }); + const users = new Map(corpus.seededUsers.map((user) => [user.name, user])); + const generator = offContext.app.get(EmbeddingGeneratorService); + await generateCorpusEmbeddings(offContext.prisma, generator, corpus); + await offContext.prisma.$executeRawUnsafe( + ` + CREATE TABLE ${SNAPSHOT} AS + SELECT id, retrieval_count, used_count, unused_count, + last_retrieved_at, last_used_at + FROM memories + WHERE user_id IN (${corpus.seededUsers.map((_, i) => `$${i + 1}`).join(',')}) + `, + ...corpus.seededUsers.map((user) => user.userId), + ); + snapshotCreated = true; + const off = await runArm('diversity-off', offContext, users); + + configureArm(true); + onContext = await createTestApp(false); + await assertIsolatedLocalDatabase(onContext); + const on = await runArm('diversity-on', onContext, users); + const comparison = compare(off, on); + const artifact = { + generatedAt: new Date().toISOString(), + config: { + finalLimit: FINAL_LIMIT, + baseFlags: { + RECALL_RERANK_SCALE_FIX: true, + RECALL_LEXICAL_COVERAGE_FLOOR: true, + RECALL_RESCUE_SQL_TIEBREAK: true, + }, + diversityOff: {}, + diversityOn: { + RECALL_CANDIDATE_POOL_DEPTH: 12, + RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT: 2, + jaccardThreshold: 0.9, + }, + snapshotIsolation: 'usage restored before every query and repeat', + fixtureIntegrity, + embeddingModel: generator.getModelName(), + embeddingDimensions: generator.getDimensions(), + }, + off, + on, + comparison, + }; + writeFileSync(OUTPUT_PATH, JSON.stringify(artifact, null, 2)); + console.log( + JSON.stringify({ off: off.summary, on: on.summary, comparison }, null, 2), + ); + } finally { + const cleanupContext = onContext ?? offContext; + if (snapshotCreated && cleanupContext) { + await cleanupContext.prisma + .$executeRawUnsafe(`DROP TABLE IF EXISTS ${SNAPSHOT}`) + .catch(() => undefined); + } + if (corpus) await corpus.cleanup().catch(() => undefined); + if (onContext) await onContext.app.close().catch(() => undefined); + if (offContext) await offContext.app.close().catch(() => undefined); + } +} + +main().catch((error) => { + console.error(error); + process.exit(1); +}); diff --git a/scripts/research/score-candidate-matrix.mjs b/scripts/research/score-candidate-matrix.mjs new file mode 100755 index 0000000..41b5fcd --- /dev/null +++ b/scripts/research/score-candidate-matrix.mjs @@ -0,0 +1,94 @@ +#!/usr/bin/env node +import { readFileSync } from 'node:fs'; + +const files = process.argv.slice(2); +if (files.length === 0) { + process.stderr.write( + 'usage: score-candidate-matrix.mjs [...]\n', + ); + process.exit(2); +} + +function percentile(values, p) { + if (values.length === 0) return 0; + const sorted = [...values].sort((a, b) => a - b); + return sorted[Math.min(sorted.length - 1, Math.floor(p * sorted.length))]; +} + +const artifacts = files.map((file) => ({ + file, + ...JSON.parse(readFileSync(file, 'utf8')), +})); + +const rows = artifacts.map(({ file, summary, rows: tasks }) => { + const n = tasks.length; + const at = (k) => + tasks.filter((task) => task.rankInPage >= 0 && task.rankInPage < k).length; + const mrr = + tasks.reduce( + (sum, task) => + sum + (task.rankInPage >= 0 ? 1 / (task.rankInPage + 1) : 0), + 0, + ) / n; + const latencies = tasks + .map((task) => task.pageLatencyMs) + .filter(Number.isFinite); + return { + artifact: file, + arm: summary.label, + n, + goldAt1: at(1), + goldAt5: at(5), + goldAt10: at(10), + mrrAt10: Number(mrr.toFixed(4)), + pageSizes: [...new Set(tasks.map((task) => task.pageSize))].join(','), + poolSizes: [...new Set(tasks.map((task) => task.poolSize))].join(','), + latencyP50Ms: percentile(latencies, 0.5), + latencyP95Ms: percentile(latencies, 0.95), + }; +}); + +console.table(rows); + +const byLogicalArm = new Map(); +for (const artifact of artifacts) { + const logical = artifact.summary.label.replace(/-r[0-9]+$/, ''); + const prior = byLogicalArm.get(logical); + const signature = artifact.rows.map((task) => + task.top.map((hit) => hit.id).join(','), + ); + if (!prior) { + byLogicalArm.set(logical, signature); + continue; + } + let changedQueries = 0; + for (let i = 0; i < signature.length; i++) { + if (signature[i] !== prior[i]) changedQueries++; + } + process.stdout.write( + `${logical} stability: ${changedQueries}/20 repeated-query rankings changed\n`, + ); +} + +const formerMisses = new Set([ + 'paginated-list-endpoint', + 'webhook-processor', + 'inventory-reservation', + 'currency-conversion', + 'tenant-scoped-projects', + 'order-dto-mapping', +]); +for (const artifact of artifacts.filter((item) => + /-r1$/.test(item.summary.label), +)) { + process.stdout.write( + `\n${artifact.summary.label} former-miss ranks (0-based):\n`, + ); + for (const task of artifact.rows.filter((item) => + formerMisses.has(item.task), + )) { + process.stdout.write( + ` ${task.task}: page=${task.rankInPage} pool=${task.rankInPool} poolSize=${task.poolSize}\n`, + ); + } +} diff --git a/src/memory/memory-diversity.util.spec.ts b/src/memory/memory-diversity.util.spec.ts new file mode 100644 index 0000000..f1fa3a8 --- /dev/null +++ b/src/memory/memory-diversity.util.spec.ts @@ -0,0 +1,86 @@ +import { MemoryWithScore } from './memory.types'; +import { selectNearDuplicateDiverse } from './memory-diversity.util'; + +function candidate(id: string, raw: string, score: number): MemoryWithScore { + return { id, raw, score } as MemoryWithScore; +} + +describe('selectNearDuplicateDiverse (research)', () => { + const prefix = + 'MNEMON benchmark project fact for pagination conventions for list endpoints:'; + const decoy = (n: number) => + candidate( + `decoy-${n}`, + `${prefix} archived neighboring-service note ${n}. This historical contract belongs to legacy-paginated-list-endpoint-${n}, not the current service. Its imports, units, key formats, argument order, return types, and async behavior may differ from the current project.`, + 1 - n / 100, + ); + const gold = candidate( + 'gold', + `${prefix} import db plus User from ../lib/db.js and parsePageParams from ../lib/pagination.js. paginate should delegate directly to parsePageParams. db.queryUsers(page, pageSize) returns Promise with items and total; listUsers must merge the result with the validated 1-based page and pageSize.`, + 0.8, + ); + + it('caps almost-identical decoys and admits the distinct current contract', () => { + const ranked = [ + ...Array.from({ length: 10 }, (_, i) => decoy(i + 1)), + gold, + ]; + const selected = selectNearDuplicateDiverse(ranked, 10, 2); + + expect(selected).toHaveLength(10); + expect(selected.slice(0, 3).map((row) => row.id)).toEqual([ + 'decoy-1', + 'decoy-2', + 'gold', + ]); + }); + + it('does not cluster records that only share a topic prefix', () => { + const distinct = [ + candidate( + 'a', + `${prefix} use cursor pagination and return nextCursor.`, + 1, + ), + candidate( + 'b', + `${prefix} use offset pagination and return totalPages.`, + 0.9, + ), + candidate( + 'c', + `${prefix} use keyset pagination ordered by createdAt.`, + 0.8, + ), + ]; + + expect( + selectNearDuplicateDiverse(distinct, 3, 1).map((row) => row.id), + ).toEqual(['a', 'b', 'c']); + }); + + it('preserves incoming relevance order among selected records', () => { + const selected = selectNearDuplicateDiverse( + [decoy(1), decoy(2), decoy(3), gold], + 10, + 2, + ); + expect(selected.map((row) => row.id)).toEqual([ + 'decoy-1', + 'decoy-2', + 'gold', + 'decoy-3', + ]); + }); + + it('never exceeds the requested limit when suppressed rows exist', () => { + const ranked = [decoy(1), decoy(2), decoy(3), gold]; + const selected = selectNearDuplicateDiverse(ranked, 3, 2); + expect(selected).toHaveLength(3); + expect(selected.map((row) => row.id)).toEqual([ + 'decoy-1', + 'decoy-2', + 'gold', + ]); + }); +}); diff --git a/src/memory/memory-diversity.util.ts b/src/memory/memory-diversity.util.ts new file mode 100644 index 0000000..984e0cd --- /dev/null +++ b/src/memory/memory-diversity.util.ts @@ -0,0 +1,69 @@ +import { MemoryWithScore } from './memory.types'; + +/** + * Research-only near-duplicate diversification. + * + * Token-set Jaccard is intentionally conservative and transparent. At the + * default 0.9 threshold it clusters records that are almost the same text, + * while leaving facts that merely share a topic/prefix in separate clusters. + * Candidates retain their incoming relevance order; the only intervention is + * a cap on how many members any one near-duplicate cluster may consume. + */ +export function selectNearDuplicateDiverse( + candidates: readonly MemoryWithScore[], + limit: number, + perCluster: number, + threshold = 0.9, +): MemoryWithScore[] { + if (limit <= 0 || perCluster <= 0) return []; + + const clusters: Array<{ + representative: Set; + count: number; + }> = []; + const selected: MemoryWithScore[] = []; + const suppressed: MemoryWithScore[] = []; + + for (const candidate of candidates) { + const tokens = tokenSet(candidate.raw ?? ''); + const cluster = clusters.find( + (entry) => jaccard(tokens, entry.representative) >= threshold, + ); + + if (cluster) { + if (cluster.count >= perCluster) { + suppressed.push(candidate); + continue; + } + cluster.count++; + } else { + clusters.push({ representative: tokens, count: 1 }); + } + + selected.push(candidate); + if (selected.length >= limit) break; + } + + // Diversity must not silently shrink a caller's requested page. Once every + // available cluster has had a fair chance, backfill with suppressed rows in + // their original relevance order. + for (const candidate of suppressed) { + if (selected.length >= limit) break; + selected.push(candidate); + } + + return selected; +} + +function tokenSet(text: string): Set { + return new Set(text.toLowerCase().match(/[a-z0-9]+/g) ?? []); +} + +function jaccard(a: ReadonlySet, b: ReadonlySet): number { + if (a.size === 0 && b.size === 0) return 1; + let intersection = 0; + for (const token of a) { + if (b.has(token)) intersection++; + } + return intersection / (a.size + b.size - intersection); +} diff --git a/src/memory/memory-query-scale-fix.spec.ts b/src/memory/memory-query-scale-fix.spec.ts index eb14b21..38666ab 100644 --- a/src/memory/memory-query-scale-fix.spec.ts +++ b/src/memory/memory-query-scale-fix.spec.ts @@ -31,6 +31,7 @@ function candidateId(i: number): string { interface Harness { service: MemoryQueryService; prisma: any; + rankingService: MemoryQueryRankingService; } /** @@ -57,10 +58,15 @@ function fakeReranker(): any { * (the tail of the pool), which is the situation the corpus actually exhibits: * off-topic distractors that happen to restate the query's vocabulary. */ -function harness(opts: { reranker?: boolean } = {}): Harness { +function harness(opts: { reranker?: boolean; clustered?: boolean } = {}): Harness { const memories = Array.from({ length: POOL_SIZE }, (_, i) => ({ id: candidateId(i), - raw: `memory ${i} about ${QUERY}`, + raw: + opts.clustered && i !== 10 + ? `memory ${i} ${QUERY} archived neighboring service contract with imports units key formats argument order return types async behavior validation pagination metadata and response semantics` + : i === 10 && opts.clustered + ? `memory ${i} current contract: db.queryUsers returns the validated page and pageSize with items and total` + : `memory ${i} about ${QUERY}`, userId: 'user-1', importanceScore: 0.5, effectiveScore: 0.5, @@ -140,7 +146,7 @@ function harness(opts: { reranker?: boolean } = {}): Harness { contextService, ); - return { service, prisma }; + return { service, prisma, rankingService }; } async function page( @@ -255,6 +261,73 @@ describe('recall ranking: limit-monotonicity (RECALL_RERANK_SCALE_FIX)', () => { expect(deep).toContain(candidateId(i)); } }); + + describe('candidate depth and near-duplicate diversity wiring', () => { + it('treats configured depth as a floor and never truncates limit > depth', async () => { + process.env.RECALL_CANDIDATE_POOL_DEPTH = '12'; + const { service, rankingService } = harness({ reranker: true }); + const rerank = jest.spyOn(rankingService, 'applyReranking'); + + expect(await pageIds(service, 20)).toHaveLength(20); + expect((rerank.mock.calls[0]?.[0] as unknown[]).length).toBe(20); + }); + + it('uses configured surplus depth when limit <= depth', async () => { + process.env.RECALL_CANDIDATE_POOL_DEPTH = '12'; + const { service, rankingService } = harness({ reranker: true }); + const rerank = jest.spyOn(rankingService, 'applyReranking'); + + expect(await pageIds(service, 5)).toHaveLength(5); + expect((rerank.mock.calls[0]?.[0] as unknown[]).length).toBe(12); + }); + + it('keeps depth filtering sticky: rescue cannot re-add rows outside the bounded pool', async () => { + process.env.RECALL_CANDIDATE_POOL_DEPTH = '12'; + const { service } = harness({ reranker: true }); + + const ids = await pageIds(service, 10); + expect(ids).toHaveLength(10); + // Raw rescue ordering puts m030..m039 plus semantic m000/m001 in the + // 12-row pool. The sticky re-add may retain those rows, but must not + // expand beyond that bounded set. + expect(ids).toEqual([ + candidateId(0), + candidateId(1), + ...Array.from({ length: 8 }, (_, i) => candidateId(30 + i)), + ]); + }); + + it('supports the cluster flag independently with a bounded scan and full backfill', async () => { + delete process.env.RECALL_CANDIDATE_POOL_DEPTH; + process.env.RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT = '2'; + const { service, rankingService } = harness({ + reranker: true, + clustered: true, + }); + const rerank = jest.spyOn(rankingService, 'applyReranking'); + + const ids = await pageIds(service, 10); + expect(ids).toHaveLength(10); + expect(ids.slice(0, 3)).toEqual([ + candidateId(0), + candidateId(1), + candidateId(10), + ]); + // Pool is 40, below the explicit cluster-only scan bound of 50. + expect((rerank.mock.calls[0]?.[0] as unknown[]).length).toBe(40); + }); + }); + }); + + it('ignores candidate controls without their scale-fix dependency', async () => { + delete process.env.RECALL_RERANK_SCALE_FIX; + process.env.RECALL_CANDIDATE_POOL_DEPTH = '12'; + process.env.RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT = '2'; + const { service, rankingService } = harness({ reranker: true }); + const rerank = jest.spyOn(rankingService, 'applyReranking'); + + await pageIds(service, 5); + expect((rerank.mock.calls[0]?.[0] as unknown[]).length).toBe(POOL_SIZE); }); describe('RECALL_NO_RESCUE=true (vector-only control)', () => { diff --git a/src/memory/memory-query.service.ts b/src/memory/memory-query.service.ts index 8293480..118cea5 100644 --- a/src/memory/memory-query.service.ts +++ b/src/memory/memory-query.service.ts @@ -49,6 +49,10 @@ import { meetsLexicalCoverageFloor, ILIKE_RESCUE_BAND_BOTTOM, } from './memory-ranking.util'; +import { selectNearDuplicateDiverse } from './memory-diversity.util'; + +const MAX_CONFIGURED_CANDIDATE_POOL_DEPTH = 200; +const CLUSTER_ONLY_SCAN_DEPTH = 50; @Injectable() export class MemoryQueryService { @@ -120,6 +124,48 @@ export class MemoryQueryService { private readonly rescueSqlTiebreak = process.env.RECALL_RESCUE_SQL_TIEBREAK === 'true'; + /** + * RESEARCH FLAGS — default OFF. These decouple candidate depth from the + * caller-visible final top-K so depth/diversity can be ablated safely. + */ + private readonly candidatePoolDepth = this.positiveEnvInt( + 'RECALL_CANDIDATE_POOL_DEPTH', + ); + private readonly nearDuplicateClusterLimit = this.positiveEnvInt( + 'RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT', + ); + + /** + * Candidate controls depend on the scale fix because diversity must operate + * on one consistently-scaled total order. Configured depth is a minimum pool + * size, never a cap below the caller's requested page. Cluster-only mode gets + * a bounded 50-row scan so it stays independently useful without processing + * an unbounded rescue pool. + */ + private effectiveCandidatePoolDepth(limit: number): number | undefined { + if (!this.rerankScaleFix) return undefined; + if (this.candidatePoolDepth) { + return Math.max( + limit, + Math.min( + this.candidatePoolDepth, + MAX_CONFIGURED_CANDIDATE_POOL_DEPTH, + ), + ); + } + if (this.nearDuplicateClusterLimit) { + return Math.max(limit, CLUSTER_ONLY_SCAN_DEPTH); + } + return undefined; + } + + private positiveEnvInt(name: string): number | undefined { + const raw = process.env[name]; + if (!raw) return undefined; + const value = Number(raw); + return Number.isInteger(value) && value > 0 ? value : undefined; + } + /** * `, ASC` fragment appended to rescue `ORDER BY` clauses, or the empty * string when {@link rescueSqlTiebreak} is off — so flag-off SQL is @@ -904,15 +950,24 @@ export class MemoryQueryService { // ── ENG-29: Cross-Encoder Reranking ────────────────────────── const rerankQuery = hasTemporalIntent ? dto.query : searchQuery; + // Research depth control: preserve final `limit`, but cap the candidates + // exposed to reranking. Unset means byte-for-byte existing behaviour. + const effectiveCandidateDepth = this.effectiveCandidatePoolDepth(limit); + const rerankCandidates = effectiveCandidateDepth + ? scoredMemories.slice(0, effectiveCandidateDepth) + : scoredMemories; + const rerankCandidateIds = effectiveCandidateDepth + ? new Set(rerankCandidates.map((memory) => memory.id)) + : undefined; // Scale-fix mode ranks the entire pool rather than truncating at `limit`. // The reranker already scores every candidate (`candidates = memories`); // `limit` only controlled the final `slice`. Ranking the whole pool is what // makes the sticky re-add a no-op and the page prefix-stable in `limit`. const rerankDepth = this.rerankScaleFix - ? Math.max(limit, scoredMemories.length) + ? Math.max(limit, rerankCandidates.length) : limit; scoredMemories = await this.rankingService.applyReranking( - scoredMemories, + rerankCandidates, rerankQuery, rerankDepth, ); @@ -944,7 +999,11 @@ export class MemoryQueryService { Number.POSITIVE_INFINITY, ); const missingKeywordHits = [...keywordRescueMap.entries()] - .filter(([id]) => !scoredMemories.some((m) => m.id === id)) + .filter( + ([id]) => + (!rerankCandidateIds || rerankCandidateIds.has(id)) && + !scoredMemories.some((m) => m.id === id), + ) .map(([, mem], i) => { if (!this.rerankScaleFix) { return { @@ -968,14 +1027,20 @@ export class MemoryQueryService { // pool depth above, so this is the single point where `limit` is applied. // Truncating a total order (compareByRankKeys) at K is what gives the // prefix property across different values of `limit`. - scoredMemories = ( + const consistentlyScaled = ( missingKeywordHits.length > 0 ? [...missingKeywordHits, ...scoredMemories] : scoredMemories ) .slice() - .sort(compareByRankKeys) - .slice(0, limit); + .sort(compareByRankKeys); + scoredMemories = this.nearDuplicateClusterLimit + ? selectNearDuplicateDiverse( + consistentlyScaled, + limit, + this.nearDuplicateClusterLimit, + ) + : consistentlyScaled.slice(0, limit); } else if (missingKeywordHits.length > 0) { scoredMemories = [...missingKeywordHits, ...scoredMemories] .sort(compareByRankKeys)