Skip to content

[WIP] Fix OpenClaw plugin reporting constant score for search results - #241

Draft
guolvlin-cn with Copilot wants to merge 1 commit into
mainfrom
copilot/fix-openclaw-constant-score
Draft

[WIP] Fix OpenClaw plugin reporting constant score for search results#241
guolvlin-cn with Copilot wants to merge 1 commit into
mainfrom
copilot/fix-openclaw-constant-score

Conversation

Copilot AI commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.


This section details on the original issue you should resolve

<issue_title>[Bug]: OpenClaw plugin reports constant score 0.5 for all search results (retrieval_score dropped)</issue_title>
<issue_description>## Summary

The OpenClaw plugin (@matrixorigin/thememoria) discards the API's real relevance score and reports a constant 0.5 for every search result. The API itself is fine — this is purely a field-mapping bug in the plugin.

Consequence: any recall/relevance verification done through the OpenClaw memory_search tool is meaningless. Results appear unranked, and there is no way for the caller to tell a strong match from an unrelated one.

While confirming this I also found the limit parameter is ignored by POST /v1/memories/search (secondary issue, see below).

Environment

  • Plugin: @matrixorigin/thememoria 0.4.3 (backend: "api", apiUrl: https://api.thememoria.ai)
  • Host: OpenClaw 2026.8.1, node v26.7.0, macOS 26.6.2 arm64

Root cause

POST /v1/memories/search returns a per-item field named retrieval_score. The plugin never reads it.

openclaw/client.ts parses only a field called confidence, which the search response does not contain, so it is always null:

// openclaw/client.ts:99
confidence:
  typeof value.confidence === "number" && Number.isFinite(value.confidence)
    ? value.confidence
    : null,

openclaw/index.ts then maps that null through a normalizer whose fallback is the literal 0.5:

// openclaw/index.ts:359
function normalizeScore(confidence?: number | null): number {
  if (typeof confidence !== "number" || !Number.isFinite(confidence)) {
    return 0.5;   // <-- every search result lands here
  }
  ...
}

// openclaw/index.ts:401
score: normalizeScore(memory.confidence),

So score is 0.5 for 100% of results, always.

Two separate defects are stacked here:

  1. Wrong source field. Relevance comes from retrieval_score; confidence is a different concept (the response carries initial_confidence, a per-memory property unrelated to the query).
  2. Silent fallback to a plausible-looking constant. 0.5 is indistinguishable from a genuine mid-relevance score, so the failure is invisible. A missing score should surface as undefined/absent, not as a fabricated mid-range number.

Reproduction

Raw API call — real, well-separated scores:

curl -s -X POST "https://api.thememoria.ai/v1/memories/search" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MEMORIA_API_KEY" \
  -d '{"query":"ripgrep ignore rules hidden files false negative","top_k":10}' \
| jq -r '.[] | "\(.retrieval_score)  \(.memory_id)"'
3.0664807326043806  019f27d3079373728942af6a4bf239ea
2.198475657498051   01a028b26add7b31b9a33faf50cdeb84
2.1921523899072697  01a023e3664b70418058f8dd45b388ec
2.029844700443825   01a05e7a0e33794290a8cfb758e4ecac
1.8586899208618903  01a047e33fdb79e18ef14eb0ab5ee709
1.8210298144414723  01a039550756...
1.7811247370235601  01a01fdc8424...
1.7108728133920021  01a0226805337e92b599c9f529ba7647
1.6108306094871974  019df1b0023a...
1.5904648443268936  01a034579e2e79c1bd431d2020822bc6

Same query through the OpenClaw memory_search tool — identical ordering, all scores flattened:

score: 0.5   (result 1)
score: 0.5   (result 2)
score: 0.5   (result 3)
...
score: 0.5   (result 10)

Note the response item keys confirm no confidence field exists:

['content', 'created_at', 'initial_confidence', 'is_active', 'memory_id',
 'memory_type', 'observed_at', 'retrieval_score', 'session_id',
 'trust_tier', 'user_id']

Impact

This silently breaks any workflow that reasons about relevance:

  • Recall verification is unusable. A common practice after writing a memory is to query it back and require a top-N hit. With a constant score the caller cannot distinguish "the entry is poorly written" from "the ranking signal is gone", and will waste time rewriting content that was never the problem.
  • Threshold filtering is impossible. Any "keep results above X" logic either passes everything or nothing.
  • Callers cannot detect the degradation. 0.5 looks like a legitimate value. Compare with null/absent, which would be an obvious signal.

Suggested fix

In openclaw/client.ts, parse retrieval_score (keeping confidence as a fallback for other endpoints):

retrieval_score:
  typeof value.retrieval_score === "number" && Number.isFinite(value.retrieval_score)
    ? value.retrieval_score
    : null,

In openclaw/index.ts, prefer it and stop fabricating a value:

score: memory.retrieval_score ?? memory.confidence ?? undefined,

Two design points worth considering:

  • retrieval_score is unbounded (observed >3.0) while the plugin's score contract appears to be [0,1]. Either widen the contract or normalize explicitly — but normalization must preserve relative ordering and gaps, not collapse them.
  • Whatever the choice, an unknown score should be reported as absent rather than as a constant. Fabricating 0.5 converts a detectable failure into a silent one, which is what made this bug survive.

Secondary: limit is ignored by /v1/memories/search

Found while reproducing the above. limit has no effect; top_k works.

for L in 3 5 20; do
  n=$(curl -s -X POST "https://api.thememoria.ai/v1/memories/search" \
    -H "Content-Type: application/json" -H "Authorization: Bearer $MEMORIA_API_KEY" \
    -d "{\"query\":\"ripgrep ignore rules hidden files\",\"limit\":$L}" | jq 'length')
  echo "limit=$L -> $n"
done
# limit=3  -> 10
# limit=5  -> 10
# limit=20 -> 10

for L in 3 20; do
  n=$(curl -s -X POST "https://api.thememoria.ai/v1/memories/search" \
    -H "Content-Type: application/json" -H "Authorization: Bearer $MEMORIA_API_KEY" \
    -d "{\"query\":\"ripgrep ignore rules hidden files\",\"top_k\":$L}" | jq 'length')
  echo "top_k=$L -> $n"
done
# top_k=3  -> 3
# top_k=20 -> 20

limit should either be honoured as an alias of top_k or rejected as an unknown parameter. Silently ignoring it while defaulting to 10 is the same failure mode as the score bug: the caller gets a plausible-looking result and no signal that their parameter did nothing.

Happy to split the second one into its own issue if you prefer.
</issue_description>

<agent_instructions>aptend</agent_instructions>

Comments on the Issue (you are @copilot in this section)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: OpenClaw plugin reports constant score 0.5 for all search results (retrieval_score dropped)

2 participants