[BUG] term query on a derived field defined in the search request returns partial results non-deterministically

[eugenio-gastelum]

Describe the bug

A term query against a derived field defined in the search request body returns a partial result set, non-deterministically. The same query, repeated against an unchanged, settled, force-merged index, alternates between the correct hit count and a smaller one.

The failure is silent. Every response reports "_shards": {"total": 2, "successful": 2, "skipped": 0, "failed": 0} and "timed_out": false, and nothing is raised. A caller cannot distinguish a short answer from a true one.

Declaring the identical script as a derived field in the index mapping instead is stable across the same runs. So is a terms aggregation over a request-body derived field, and so is a term query on an ordinary indexed field over the same documents. The only combination that fails is term query + request-body derived field.

This looks related to but distinct from:

• [BUG] Derived fields not resolved during can_match phase #20965 (can_match phase) — that is reproducible here too, but the failures below occur at default settings with "skipped": 0, so shard skipping is not what is happening.
• [BUG] Cannot query derived fields directlly by defining them on search request on OS-2.15 #16487 — that is about a derived field referenced only in the fetch fields block. Here the field is used in a query clause.

Related component

Search

To Reproduce

Save as repro.sh and run with OS=http://localhost:9200 ./repro.sh. It creates a 2-shard index with 20 matching and 8 non-matching documents, force-merges, waits for the index to settle, then issues the same query 100 times in three variants.

#!/usr/bin/env bash
set -u

OS="${OS:-http://localhost:9200}"
IDX=derived-repro
RUNS="${RUNS:-100}"
EXPECTED=20

curl -s -X DELETE "$OS/$IDX" >/dev/null

curl -s -X PUT "$OS/$IDX" -H 'Content-Type: application/json' -d '{
  "settings": { "number_of_shards": 2, "number_of_replicas": 0 },
  "mappings": {
    "properties": {
      "source_value": { "type": "boolean", "index": false, "doc_values": false },
      "plain_keyword": { "type": "keyword" }
    },
    "derived": {
      "mapped_state": {
        "type": "keyword",
        "script": { "source": "def v = params._source.source_value; if (v == null) { emit(\"missing\"); } else if ((boolean) v) { emit(\"yes\"); } else { emit(\"no\"); }" }
      }
    }
  }
}' >/dev/null

BULK=""
for i in $(seq 1 20); do
  BULK="$BULK{\"index\":{\"_id\":\"m$i\"}}\n{\"source_value\":true,\"plain_keyword\":\"yes\"}\n"
done
for i in $(seq 1 8); do
  BULK="$BULK{\"index\":{\"_id\":\"n$i\"}}\n{\"source_value\":false,\"plain_keyword\":\"no\"}\n"
done
printf "$BULK" | curl -s -X POST "$OS/$IDX/_bulk" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- >/dev/null

curl -s -X POST "$OS/$IDX/_refresh" >/dev/null
curl -s -X POST "$OS/$IDX/_forcemerge?max_num_segments=1" >/dev/null
sleep 5

INLINE='{
  "size": 100,
  "track_total_hits": true,
  "derived": {
    "inline_state": {
      "type": "keyword",
      "script": { "source": "def v = params._source.source_value; if (v == null) { emit(\"missing\"); } else if ((boolean) v) { emit(\"yes\"); } else { emit(\"no\"); }" }
    }
  },
  "query": { "term": { "inline_state": "yes" } }
}'

MAPPED='{
  "size": 100,
  "track_total_hits": true,
  "query": { "term": { "mapped_state": "yes" } }
}'

CONTROL='{
  "size": 100,
  "track_total_hits": true,
  "query": { "term": { "plain_keyword": "yes" } }
}'

run_case() {
  local label="$1" body="$2" wrong=0 seen=""
  for _ in $(seq 1 "$RUNS"); do
    local n
    n=$(curl -s -X POST "$OS/$IDX/_search" -H 'Content-Type: application/json' \
        -d "$body" | python3 -c 'import json,sys; print(json.load(sys.stdin)["hits"]["total"]["value"])')
    [ "$n" != "$EXPECTED" ] && wrong=$((wrong + 1))
    case " $seen " in *" $n "*) ;; *) seen="$seen $n";; esac
  done
  printf '%-46s wrong %2d/%-3d  counts seen:%s\n' "$label" "$wrong" "$RUNS" "$seen"
}

echo "OpenSearch: $(curl -s "$OS" | python3 -c 'import json,sys; print(json.load(sys.stdin)["version"]["number"])')"
echo "Expecting $EXPECTED hits every time, over $RUNS identical runs."
echo
run_case "term on a normal keyword (control)" "$CONTROL"
run_case "term on a MAPPING-declared derived field" "$MAPPED"
run_case "term on a REQUEST-BODY derived field" "$INLINE"

echo
echo "Shard report from one request-body run:"
curl -s -X POST "$OS/$IDX/_search" -H 'Content-Type: application/json' -d "$INLINE" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print("  hits:", d["hits"]["total"]["value"], " _shards:", d["_shards"], " timed_out:", d["timed_out"])'

Expected behavior

All three variants return 20 hits on every run. The three are semantically identical: same documents, same predicate, same script.

Actual behavior

OpenSearch 3.5.0

Expecting 20 hits every time, over 100 identical runs.

term on a normal keyword (control)             wrong  0/100  counts seen: 20
term on a MAPPING-declared derived field       wrong  0/100  counts seen: 20
term on a REQUEST-BODY derived field           wrong 16/100  counts seen: 20 9

OpenSearch 3.7.0

Expecting 20 hits every time, over 100 identical runs.

term on a normal keyword (control)             wrong  0/100  counts seen: 20
term on a MAPPING-declared derived field       wrong  0/100  counts seen: 20
term on a REQUEST-BODY derived field           wrong  8/100  counts seen: 20 9

A failing response, in full shard detail:

hits: 9  _shards: {'total': 2, 'successful': 2, 'skipped': 0, 'failed': 0}  timed_out: False

Notes on the shape of the failures:

• The wrong answer is always the same value, 9 of the expected 20 — roughly one shard's worth. Consistent with one of the two shards concluding that nothing matches rather than with documents being dropped individually.
• Both versions are affected; 3.7.0 is not a fix.
• The rate varies between runs of the whole script (I have observed anywhere from 8% to roughly 48% across sessions on the same image), but the control and mapping-declared rows have never once been wrong.

Things I checked and ruled out:

• Not can_match. Failing responses report "skipped": 0. Forcing the phase with pre_filter_shard_size=1 on the same index gives a clearly different signature — "skipped": 1 and 9 hits deterministically, 5 runs out of 5 — which looks like [BUG] Derived fields not resolved during can_match phase #20965. The failure reported here is intermittent and skips nothing, so it is a separate path.
• Not index count. A single index reproduces it; it is not a cross-index or alias effect.
• Not script compilation or cache pressure. Across a loop of 60 requests in which 6 returned the wrong count, _nodes/stats/script was unchanged before and after: {'compilations': 1, 'cache_evictions': 0, 'compilation_limit_triggered': 0}. The script is compiled once and never recompiled or evicted.
• Not segment state. The index is force-merged to one segment and left to settle before the loop; the control query is stable in the same loop against the same index.
• Not aggregation-vs-search generally. A terms aggregation over the same request-body derived field is stable; only the query path is affected.

Additional context

The practical impact is that a request-body derived field cannot be used in a filter for anything where correctness matters, because there is no signal to check. No shard fails, nothing is skipped, no exception is raised, and the response is a plausible smaller result set. Any caller that treats the response as an answer will silently act on incomplete data.

The workaround is to declare the derived field in the index mapping instead. That is additive — PUT /<index>/_mapping with a derived block is accepted on an index already holding documents, and since the script reads _source at query time it applies to documents indexed before the declaration, so no reindex is needed. Worth noting because the documentation presents the two forms as equivalent, and on this evidence they are not.

Host/Environment

• OpenSearch: 3.5.0 and 3.7.0, official opensearchproject/opensearch Docker images, single node, security plugin disabled.
• Index: 2 primary shards, 0 replicas.
• Host: macOS on aarch64, Docker 28.4.0.