SIGILL in native cosine vectorSearch on older x86_64 CPU (no F16C/FMA) with @lancedb/lancedb

[lyz9273]

Plugin Version

1.1.0 beta10

OpenClaw Version

2026.3.13

Bug Description

Summary

On an older x86_64 CPU, LanceDB native vector search crashes the Node process with SIGILL when
using cosine distance.

environment:

• table.vectorSearch(query).toArray() works
• table.vectorSearch(query).distanceType('l2').toArray() works
• table.vectorSearch(query).distanceType('dot').toArray() works
• table.vectorSearch(query).distanceType('cosine').toArray() crashes the process with SIGILL

This is not a JS exception. The process exits immediately due to an illegal instruction.

Environment

• OS: Ubuntu Server
• Node: v25.8.1
• Package: @lancedb/lancedb@0.26.2
• Native addon:
  • @lancedb/lancedb-linux-x64-gnu
• CPU:
  • Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz

lscpu flags include:

• avx
• sse4_1
• sse4_2

But do not include:

• avx2
• f16c
• fma

Actual behavior

The process exits with SIGILL.

I reproduced it from a child process and got:

{
"status": null,
"signal": "SIGILL"
}

What I found

I inspected the native addon binary and found these local symbols:

• cosine_f16_avx2
• dot_f16_avx2
• norm_l2_f16_avx2

Disassembly of cosine_f16_avx2 shows instructions such as:

• vcvtph2ps
• vfmadd231ps
• vfmadd231ss

These require CPU features that are not present on this machine:

• vcvtph2ps -> f16c
• vfmadd* -> fma

That matches the observed crash very closely.

Additional observations

These combinations succeed:

• vectorSearch(...).toArray()
• vectorSearch(...).distanceType("l2").toArray()
• vectorSearch(...).distanceType("dot").toArray()

This combination crashes:

• vectorSearch(...).distanceType("cosine").toArray()

So this appears specific to the native cosine path.

Workaround

In my application/plugin, I worked around this by avoiding LanceDB's native cosine path and
instead:

1. reading candidate rows through the normal query API
2. computing cosine similarity in JS
3. sorting/filtering in application code

That avoids the crash, but obviously it is slower and not ideal.

Question

Is this a known CPU compatibility issue in the prebuilt binary for @lancedb/lancedb@0.26.2?

If not, I suspect one of these is happening:

• the cosine path is compiled assuming f16c / fma
• or runtime dispatch is selecting that path on unsupported CPUs

If useful, I can provide more details from objdump, lscpu, or a reduced repro.

Expected Behavior

Either:

• cosine vector search should work on CPUs without AVX2/F16C/FMA, or
• the library should detect unsupported CPU features and fall back safely, or
• it should fail with a normal runtime error instead of killing the process

Steps to Reproduce

Minimal reproduction

  js
  const lancedb = require("@lancedb/lancedb");

  (async () => {
    const db = await lancedb.connect("/path/to/db");
    const table = await db.openTable("memories");

    const rows = await table.query().select(["vector"]).limit(1).toArray();
    const q = Array.from(rows[0].vector);

    console.log("before cosine");
    await table.vectorSearch(q).distanceType("cosine").limit(1).toArray();
    console.log("after cosine");
  })();

Error Logs / Screenshots

Embedding Provider

OpenAI

OS / Platform

Ubuntu Server

[AliceLJY]

Hi @lyz9273, thanks for the detailed investigation — the distance-type isolation test is really helpful.

This is a LanceDB native library issue: the cosine distance kernel uses F16C/FMA SIMD instructions that aren't available on your CPU. L2 and dot product use different code paths, which is why they work.

This is outside memory-lancedb-pro's scope — we don't control LanceDB's native SIMD dispatch.

Possible workaround (if you want to fork)

You could change the distance type to l2 in the vector search calls in src/store.ts. Search for .distanceType("cosine") and replace with .distanceType("l2"). This trades some retrieval quality for CPU compatibility.

Upstream

You may want to file this with LanceDB directly — they could add runtime CPU feature detection to fall back gracefully instead of SIGILL.