Move off deprecated reranker and LLM endpoints

[karthikt-nvidia]

Moves this branch off two deprecated inference endpoints, and validates the result on a GPU.

Builds on the work in #43, which targeted main. This one targets dev/streaming-data-to-rag directly — the branch the Streaming Data to RAG blueprint actually pins — so it can be reviewed and merged on its own. Opening it separately rather than pushing to #43 because I don't have write access to that branch.

Why

The reranker is already gone. The hosted llama-3_2-nv-rerankqa-1b-v2 endpoint this branch calls returns:

[410] Gone — This endpoint has reached its end of life on 2026-05-18T00:00:00Z and is no longer available.

So retrieval reranking fails on this branch today, not at some future date.

The LLM is next. nvidia/nvidia-nemotron-nano-9b-v2 (used for chat, summarization and notification) is in the August 2026 deprecation wave. It still responds, but not for long.

What changed

Component	Before	After
Reranker	llama-3.2-nv-rerankqa-1b-v2 @ ai.api.nvidia.com	llama-nemotron-rerank-vl-1b-v2, self-hosted
LLM (×3)	nvidia-nemotron-nano-9b-v2 @ integrate.api	nemotron-3-nano-30b-a3b @ integrate.api
Embedding	unchanged	unchanged
Parakeet ASR	unchanged	unchanged

The reranker has to be self-hosted — this isn't a style preference

Pointing the config at the hosted successor does not work with the pinned SDK. langchain-nvidia-ai-endpoints==0.3.7 validates rerank model names against its own registry:

Model nvidia/llama-nemotron-rerank-vl-1b-v2 is unknown, check `available_models`

That registry knows only four models, all previous-generation. The request is rejected client-side before it's ever sent — note the hosted endpoint itself answers 200 to a raw curl, so this fails only through the SDK.

Upgrading the SDK isn't an option either:

ERROR: Cannot install langchain-nvidia-ai-endpoints==0.3.19 and langchain_core==0.3.21
because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible

So the reranker follows the pattern this branch already uses for embedding: a reranker-nim service in compose.yaml (mirroring embedding-nim), with base_url pointing at it. RERANKER_GPU_ID defaults to 0 alongside EMBEDDING_GPU_ID; set them apart on multi-GPU hosts.

The LLM needed no such treatment — ChatNVIDIA passes model names straight through with no registry check. I verified that against the pinned SDK rather than assuming it, since the two clients behave differently.

Stale defaults in source

milvus_db.py and aiq/utils.py still hardcode the dead reranker as their default. config.yaml overrides them at runtime, so nothing is broken today — but they're the values a new call site inherits, so they're updated too.

What is deliberately not changed

Embedding and the Parakeet ASR NIM are both self-hosted containers on this branch. Hosted-endpoint deprecation doesn't apply to them, so they're left alone.

This is worth stating explicitly: Parakeet has been flagged as needing replacement in the deprecation tracking, and on this branch that flag is a false positive — it runs from nvcr.io, not build.nvidia.com. No action needed for it here.

Validation

Run on a single L4. The reranker NIM (llama-nemotron-rerank-vl-1b-v2:2.3.0) came up healthy and advertises the model id the config asks for:

GET /v1/models → ["nvidia/llama-nemotron-rerank-vl-1b-v2"]

Then the pinned SDK (langchain-nvidia-ai-endpoints==0.3.7, langchain_core==0.3.21) reranking through it, query "what was said about GPUs?":

1.  -5.4147   GPU acceleration for radio signal processing
2.  -7.4962   weather clear skies wind ten knots
3.  -7.9099   banana bread recipe with walnuts

Correct ordering with a clear margin — so the client accepts the model, reaches the NIM, and gets meaningful scores. That's the specific thing that fails against the hosted endpoint.

LLM: nemotron-3-nano-30b-a3b verified responding through ChatNVIDIA on the pinned SDK.

Worth flagging

• nemotron-3-nano-30b-a3b has a limited shelf life. Nemotron-3.5-Nano goes live 2026-08-11, and the 3-Nano generation is expected to retire roughly two weeks after. This change gets the branch off a model that's deprecating now, but another LLM bump is likely soon. Happy to follow up once 3.5-Nano is available.
• A longer-term option, if you'd prefer to stop tracking hosted-endpoint churn: the same self-hosting treatment applied to the reranker here works for the LLM too. Entirely your call — this PR keeps the LLM hosted to stay close to the existing shape.
• Not run end to end: I validated the reranker path and the LLM endpoint, not a full Streaming Data to RAG pipeline run with SDR ingest.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[karthikt-nvidia]

Hardware footprint validated on RTX 6000 Ada (48 GB)

Following up on the "do the locally-hosted models still fit on the target GPU" question — deployed the self-hosted NIMs on a real NVIDIA RTX 6000 Ada (48 GB) and measured.

All three self-hosted NIMs up (post-swap):

$ nvidia-smi --query-gpu=name,memory.used,memory.total --format=csv,noheader
NVIDIA RTX 6000 Ada Generation, 20448 MiB, 49140 MiB

$ nvidia-smi --query-compute-apps=used_memory --format=csv,noheader
5810 MiB     # reranker-nim   llama-nemotron-rerank-vl-1b-v2:2.3.0  (the swap adds only this)
3560 MiB     # embedding-nim  llama-3.2-nv-embedqa-1b-v2:1.10.0
11056 MiB    # parakeet-nim   parakeet-0-6b-ctc-en-us:2  (ASR)

~20 GB of 48 GB used — ~28 GB (58%) headroom. Fits comfortably. The swap only adds the reranker (~5.7 GB); embedding and ASR were already local.

Reranker serves correctly through the pinned SDK (langchain-nvidia-ai-endpoints==0.3.7, the client that rejects the hosted successor client-side):

$ python3 - <<'PY'
from langchain_nvidia_ai_endpoints import NVIDIARerank
from langchain_core.documents import Document
r = NVIDIARerank(model="nvidia/llama-nemotron-rerank-vl-1b-v2",
                 base_url="http://reranker-nim:8000/v1", api_key="x")
docs = [Document(page_content="GPU acceleration for radio signal processing"),
        Document(page_content="a banana bread recipe")]
print([round(d.metadata["relevance_score"], 3) for d in r.compress_documents(
        query="what was said about GPUs?", documents=docs)])
PY
[-5.413, -7.808]   # GPU passage ranked first — correct

On the reranker path

A full end-to-end deploy of streaming-data-to-rag (A100, 14 containers, live ASR → ingest → graph-RAG with grounded citations) confirmed something worth flagging: the blueprint runs in graph-rag mode by default, and NVIDIARerank is only consumed on the vector-rag path.

$ docker logs vss-ctx-rag-retriever 2>&1 | grep -i 'rag type'
rag type: graph-rag

$ docker logs vss-ctx-rag-retriever 2>&1 | grep -icE '410|Gone'
0

So the dead hosted reranker (ai.api…/llama-3_2-nv-rerankqa-1b-v2 → HTTP 410, EOL 2026-05-18) is constructed-but-unused in the default config — the pipeline works despite it. This swap is therefore cleaning up real tech debt (a 410 endpoint that fails the moment anyone switches to vector-rag) rather than fixing a live outage. Wanted to be precise about that rather than overstate it.

(Also surfaced while deploying: a docker compose project-name collision that breaks the documented bring-up — worked around with -p fmstack. Happy to file that separately if useful.)

[david-duu]

Hi instead of hardcoding models we have switch to ENV in latest main

https://github.com/NVIDIA/context-aware-rag/blob/main/config/config.yaml

[david-duu]

/ok to test 88f992c