diff --git a/docs/docs/extraction/agentic-retrieval-concept.md b/docs/docs/extraction/agentic-retrieval-concept.md index 32d4d623a..d8b8fac04 100644 --- a/docs/docs/extraction/agentic-retrieval-concept.md +++ b/docs/docs/extraction/agentic-retrieval-concept.md @@ -1,10 +1,5 @@ # Agentic retrieval (concept) -!!! note - - This documentation describes NeMo Retriever Library. - - Agentic retrieval means **iterative, tool-driven** retrieval: an agent plans steps, issues searches, may refine filters, and optionally reranks until it has enough context to answer. NeMo Retriever Library focuses on document ingestion, embeddings, vector stores, hybrid search, and reranking. Orchestration frameworks call these building blocks from your application. diff --git a/docs/docs/extraction/benchmarking.md b/docs/docs/extraction/benchmarking.md index eae7b7435..c3c7d3c30 100644 --- a/docs/docs/extraction/benchmarking.md +++ b/docs/docs/extraction/benchmarking.md @@ -1,10 +1,5 @@ # NeMo Retriever Library integration testing framework -!!! note - - This documentation describes NeMo Retriever Library. - - A configurable, dataset-agnostic testing framework for end-to-end validation of NeMo Retriever Library pipelines. This framework uses structured YAML configuration for type safety, validation, and parameter management. ## Dataset Prerequisites diff --git a/docs/docs/extraction/choose-your-path.md b/docs/docs/extraction/choose-your-path.md index 9db21bc2c..1d02ab367 100644 --- a/docs/docs/extraction/choose-your-path.md +++ b/docs/docs/extraction/choose-your-path.md @@ -1,10 +1,5 @@ # Choose your path -!!! note - - This documentation describes NeMo Retriever Library. - - Use this page to pick documentation and deployment options that match your goal. ## I want to run locally or embed the library diff --git a/docs/docs/extraction/chunking.md b/docs/docs/extraction/chunking.md index 50d548e10..6cdbd1560 100644 --- a/docs/docs/extraction/chunking.md +++ b/docs/docs/extraction/chunking.md @@ -1,10 +1,5 @@ # Split Documents -!!! note - - This documentation describes NeMo Retriever Library. - - Splitting, also known as chunking, breaks large documents or text into smaller, manageable sections to improve retrieval efficiency. After chunking, only the most relevant pieces of information are retrieved for a given query. Chunking also prevents text from exceeding the context window of the embedding model. diff --git a/docs/docs/extraction/cli-reference.md b/docs/docs/extraction/cli-reference.md index 51164b358..ed07d1277 100644 --- a/docs/docs/extraction/cli-reference.md +++ b/docs/docs/extraction/cli-reference.md @@ -1,10 +1,5 @@ # CLI Reference -!!! note - - This documentation describes NeMo Retriever Library. - - After you install the Python dependencies, you can use the [NeMo Retriever Library](overview.md) command line interface (CLI). To use the CLI, use the `nemo-retriever` command. diff --git a/docs/docs/extraction/concepts.md b/docs/docs/extraction/concepts.md index 096206227..f2f85e433 100644 --- a/docs/docs/extraction/concepts.md +++ b/docs/docs/extraction/concepts.md @@ -1,10 +1,5 @@ # Concepts -!!! note - - This documentation describes NeMo Retriever Library. - - These terms appear throughout NeMo Retriever Library documentation. ## Job @@ -26,6 +21,7 @@ Optionally, the library can compute **embeddings** for extracted content and sto ## Deployment modes - **Library mode** — Run without the full container stack where appropriate ([quickstart](quickstart-library-mode.md)). +- **Docker Compose (self-hosted)** — [Container stack quickstart](quickstart-guide.md) for running the full microservices pipeline locally. - **Helm / Kubernetes** — [Helm-based deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/main/helm/README.md) for cluster operations. - **Notebooks** — [Jupyter examples](notebooks.md) for experimentation and RAG demos. diff --git a/docs/docs/extraction/content-metadata.md b/docs/docs/extraction/content-metadata.md index 332f75f1c..7f549cf51 100644 --- a/docs/docs/extraction/content-metadata.md +++ b/docs/docs/extraction/content-metadata.md @@ -8,11 +8,6 @@ The definitions used in this documentation are the following: Metadata can be extracted from a source or content, or generated by using models, heuristics, or other methods. -!!! note - - This documentation describes NeMo Retriever Library. - - ## Source File Metadata diff --git a/docs/docs/extraction/contributing.md b/docs/docs/extraction/contributing.md index eb4a02e27..6d018b4c3 100644 --- a/docs/docs/extraction/contributing.md +++ b/docs/docs/extraction/contributing.md @@ -1,9 +1,148 @@ # Contributing to NeMo Retriever Library -!!! note +External contributions will be welcome soon, and they are greatly appreciated. For repository policy, coding standards, and the contribution process, refer to **[Contributing to NeMo Retriever](https://github.com/NVIDIA/NeMo-Retriever/blob/main/CONTRIBUTING.md)** on GitHub. - This documentation describes NeMo Retriever Library. +The sections below describe how to configure your machine and Git remotes so you can work on documentation (or code) against **[NVIDIA/NeMo-Retriever](https://github.com/NVIDIA/NeMo-Retriever)** using a fork and a separate publishing clone. +--- -External contributions will be welcome soon, and they are greatly appreciated! -For more information, refer to [Contributing to NeMo Retriever](https://github.com/NVIDIA/NeMo-Retriever/blob/main/CONTRIBUTING.md). +## Set up your writing and development environment + +### SSH authentication (one time for each computer) + +1. **Create an SSH key** on your computer. Follow steps 1–3 in [Generating a new SSH key and adding it to the ssh-agent](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent). (You only need the key-generation steps; you can skip configuring ssh-agent if your organization prefers not to use it.) + +2. **Add the public key to GitHub** using [Adding a new SSH key to your GitHub account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account). + +### Commit signing for GitHub (one time for each computer) + +1. **Create a GPG key** following [Generating a new GPG key](https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key). + +2. **Tell Git which key to use:** + + ```bash + git config --global user.signingkey YOUR_KEY_ID + ``` + +3. **Sign every commit by default (recommended if your org requires signed commits):** + + ```bash + git config --global commit.gpgsign true + ``` + +4. **Optional — sign a single commit:** + + ```bash + git commit -S -m "your message" + ``` + + or + + ```bash + git commit --gpg-sign -m "your message" + ``` + +5. **Optional — skip signing for one commit:** + + ```bash + git commit --no-gpg-sign -m "Unsigned commit" + ``` + +--- + +## Set up your writing and development clone (fork) + +You do day-to-day work in a clone of **your fork**, with `upstream` pointing at NVIDIA’s repo. + +1. **Get access** to **[https://github.com/NVIDIA/NeMo-Retriever](https://github.com/NVIDIA/NeMo-Retriever)** (and permission to fork it, per your organization). + +2. **Create a fork** + + - Open the **Fork** menu, then choose **Create a new fork**. + - Accept the default repository name (`NeMo-Retriever`) unless your org requires another name. + - **Deselect** “Copy the main branch only” if you need other branches locally; you can recover later with `git fetch upstream --tags` (see below). + - Click **Create fork**. + +3. **Clone the fork** onto your machine: + + - Pick a parent folder, for example `C:\_work\NeMo-Retriever-fork` or `C:\_repositories\NeMo-Retriever-fork`. + - Open a terminal in that folder, then clone: + + ```bash + git clone git@github.com:/NeMo-Retriever.git + ``` + + - Enter the repository directory (default folder name is usually `NeMo-Retriever`): + + ```bash + cd NeMo-Retriever + ``` + + - **Add NVIDIA’s repo as `upstream`:** + + ```bash + git remote add upstream https://github.com/NVIDIA/NeMo-Retriever.git + ``` + + - If the fork was created with **only** the default branch, fetch the rest from upstream when needed: + + ```bash + git fetch upstream --tags + ``` + +Confirm remotes: + +```bash +git remote -v +``` + +You should see `origin` pointing at your fork and `upstream` at `NVIDIA/NeMo-Retriever`. + +--- + +## Set up your publishing clone (canonical repo) + +Some workflows use a **second clone** of the **official** repository (not your fork) for publishing or internal automation. + +1. Choose a **different** directory from your fork clone. On Windows, your team may require this clone inside **WSL**; follow internal guidance. + +2. Clone NVIDIA’s repository: + + ```bash + git clone git@github.com:NVIDIA/NeMo-Retriever.git + ``` + +After setup you typically have **two** working copies: one from your fork (with `upstream` configured) and one straight from `NVIDIA/NeMo-Retriever`. + +--- + +## Make a documentation change + +### Target branches + +Decide where the change lands: + +- **`main` only** +- A **release** branch only (for example `release/25.9.0`) +- **Both** `main` and a release branch — commit to `main` first, then [cherry-pick](https://git-scm.com/docs/git-cherry-pick) the commits onto the release branch. + +### Keep your fork and local clone in sync with NVIDIA + +From your **fork** clone, on each branch you care about (example uses `main`; substitute `develop` or a release branch as needed): + +```bash +git checkout main +git fetch upstream +git merge upstream/main +git push origin main +``` + +Use a **space** between the remote name and the branch: `git push origin main`. (`git push origin/main` is invalid and Git will report an error.) + +Repeat `checkout` / `fetch` / `merge` / `push` for every branch you maintain (`main`, `develop`, release branches, and so on). + +--- + +## Related + +- [Contributing to NeMo Retriever](https://github.com/NVIDIA/NeMo-Retriever/blob/main/CONTRIBUTING.md) — authoritative contribution guidelines in the repository diff --git a/docs/docs/extraction/custom-metadata.md b/docs/docs/extraction/custom-metadata.md index 51a132d21..f2d464445 100644 --- a/docs/docs/extraction/custom-metadata.md +++ b/docs/docs/extraction/custom-metadata.md @@ -1,10 +1,5 @@ # Use Custom Metadata to Filter Search Results -!!! note - - This documentation describes NeMo Retriever Library. - - You can upload custom metadata for documents during ingestion. By uploading custom metadata you can attach additional information to documents, and use it for filtering results during retrieval operations. diff --git a/docs/docs/extraction/data-store.md b/docs/docs/extraction/data-store.md index 4e714a66b..1646ed4d3 100644 --- a/docs/docs/extraction/data-store.md +++ b/docs/docs/extraction/data-store.md @@ -2,10 +2,6 @@ Use this documentation to learn how [NeMo Retriever Library](overview.md) handles and uploads data. -!!! note - - This documentation describes NeMo Retriever Library. - ## Overview diff --git a/docs/docs/extraction/embedding-nims-models.md b/docs/docs/extraction/embedding-nims-models.md index 48410f9d8..949ccc56b 100644 --- a/docs/docs/extraction/embedding-nims-models.md +++ b/docs/docs/extraction/embedding-nims-models.md @@ -1,10 +1,5 @@ # Embedding NIMs and models -!!! note - - This documentation describes NeMo Retriever Library. - - Embeddings turn extracted text and multimodal content into vectors for semantic search. NeMo Retriever Library integrates with NVIDIA NIM microservices for embedding. Model names and compatibility vary by release; refer to the [Support matrix](support-matrix.md) and the [NVIDIA NIM catalog](https://build.nvidia.com/). For multimodal or VLM embeddings, refer to [Multimodal embeddings (VLM)](vlm-embed.md). diff --git a/docs/docs/extraction/environment-config.md b/docs/docs/extraction/environment-config.md index db776e348..4f45008f4 100644 --- a/docs/docs/extraction/environment-config.md +++ b/docs/docs/extraction/environment-config.md @@ -3,10 +3,6 @@ The following are the environment variables that you can use to configure [NeMo Retriever Library](overview.md). You can specify these in your .env file or directly in your environment. -!!! note - - This documentation describes NeMo Retriever Library. - ## General Environment Variables diff --git a/docs/docs/extraction/evaluate-on-your-data.md b/docs/docs/extraction/evaluate-on-your-data.md index edede12fa..34605ed66 100644 --- a/docs/docs/extraction/evaluate-on-your-data.md +++ b/docs/docs/extraction/evaluate-on-your-data.md @@ -1,10 +1,5 @@ # Evaluate on your data -!!! note - - This documentation describes NeMo Retriever Library. - - Retrieval and ingestion performance **depend on your documents**, hardware, and pipeline settings. Use the following when measuring quality and throughput on **your** datasets. ## Benchmarking and baselines diff --git a/docs/docs/extraction/extraction-charts-infographics.md b/docs/docs/extraction/extraction-charts-infographics.md index 1e2814c70..032a0779c 100644 --- a/docs/docs/extraction/extraction-charts-infographics.md +++ b/docs/docs/extraction/extraction-charts-infographics.md @@ -1,10 +1,5 @@ # Charts and infographics -!!! note - - This documentation describes NeMo Retriever Library. - - Charts and infographic regions are classified as graphic elements and processed with the corresponding NVIDIA NIM workflows (for example, **yolox-graphic-elements** in current releases). Outputs use the same metadata schema as other extracted objects. **Related** diff --git a/docs/docs/extraction/extraction-ocr-scanned.md b/docs/docs/extraction/extraction-ocr-scanned.md index 21eabb10b..20333458b 100644 --- a/docs/docs/extraction/extraction-ocr-scanned.md +++ b/docs/docs/extraction/extraction-ocr-scanned.md @@ -1,10 +1,5 @@ # OCR and scanned documents -!!! note - - This documentation describes NeMo Retriever Library. - - Scanned PDFs and image-only pages rely on OCR and hybrid paths that combine native text extraction with OCR when needed. For extract methods such as `ocr` and `pdfium_hybrid`, refer to the [Python API reference](python-api-reference.md). **Related** diff --git a/docs/docs/extraction/extraction-tables.md b/docs/docs/extraction/extraction-tables.md index ec01624ca..32fcd9a08 100644 --- a/docs/docs/extraction/extraction-tables.md +++ b/docs/docs/extraction/extraction-tables.md @@ -1,10 +1,5 @@ # Tables -!!! note - - This documentation describes NeMo Retriever Library. - - NeMo Retriever Library detects tables as structured page elements, processes them through the appropriate NIMs, and exports formats suitable for downstream RAG (including Markdown-oriented representations where configured). Availability depends on pipeline and model configuration; refer to the [Support matrix](support-matrix.md). **Related** diff --git a/docs/docs/extraction/faq.md b/docs/docs/extraction/faq.md index 00f8dddc9..14d42d417 100644 --- a/docs/docs/extraction/faq.md +++ b/docs/docs/extraction/faq.md @@ -2,11 +2,6 @@ This documentation contains the Frequently Asked Questions (FAQ) for [NeMo Retriever Library](overview.md). -!!! note - - This documentation describes NeMo Retriever Library. - - ## What if I already have a retrieval pipeline? Can I just use NeMo Retriever Library? diff --git a/docs/docs/extraction/getting-started-about.md b/docs/docs/extraction/getting-started-about.md index bdcfd4508..2f0a2a56f 100644 --- a/docs/docs/extraction/getting-started-about.md +++ b/docs/docs/extraction/getting-started-about.md @@ -1,10 +1,5 @@ # About getting started -!!! note - - This documentation describes NeMo Retriever Library. - - This section walks you from **access and prerequisites** through **first deployment** and **hands-on notebooks**. Typical order: @@ -13,6 +8,7 @@ Typical order: 2. Confirm [Prerequisites](prerequisites.md) and the [Support matrix](support-matrix.md) for your OS, GPU, and software stack. 3. Deploy using one of: - [Library mode](quickstart-library-mode.md) (without full stack containers where appropriate) + - [Docker Compose (self-hosted)](quickstart-guide.md) for the reference microservices stack in containers - [Helm chart](https://github.com/NVIDIA/NeMo-Retriever/blob/main/helm/README.md) for Kubernetes environments 4. Explore [Jupyter Notebooks](notebooks.md) for end-to-end examples. diff --git a/docs/docs/extraction/hosted-nims-when-to-use.md b/docs/docs/extraction/hosted-nims-when-to-use.md index 02e1a249d..58a467c49 100644 --- a/docs/docs/extraction/hosted-nims-when-to-use.md +++ b/docs/docs/extraction/hosted-nims-when-to-use.md @@ -1,10 +1,5 @@ # When to use NVIDIA-hosted NIMs -!!! note - - This documentation describes NeMo Retriever Library. - - [NVIDIA-hosted NIMs](https://build.nvidia.com/) run inference on NVIDIA-managed infrastructure. You call models with API keys (refer to [Get your API key](ngc-api-key.md)) without operating GPU nodes yourself. Consider hosted NIMs when: diff --git a/docs/docs/extraction/how-to-use-this-documentation.md b/docs/docs/extraction/how-to-use-this-documentation.md index 44b94d579..7247e87dc 100644 --- a/docs/docs/extraction/how-to-use-this-documentation.md +++ b/docs/docs/extraction/how-to-use-this-documentation.md @@ -1,10 +1,5 @@ # How to use this documentation -!!! note - - This documentation describes NeMo Retriever Library. - - Use the sections below as a reading order that matches how you run NeMo Retriever Library. ## NeMo Retriever Library (local or embedded) diff --git a/docs/docs/extraction/image-captioning.md b/docs/docs/extraction/image-captioning.md index 2ab11998f..33bc087ae 100644 --- a/docs/docs/extraction/image-captioning.md +++ b/docs/docs/extraction/image-captioning.md @@ -1,10 +1,5 @@ # Image captioning -!!! note - - This documentation describes NeMo Retriever Library. - - Image captioning generates natural-language descriptions for unstructured image content. Retrieval can then use text embeddings over captions and visual embeddings where you configure them. **Related** diff --git a/docs/docs/extraction/images/overview-extraction.png b/docs/docs/extraction/images/overview-extraction.png index 31f71f214..488200d59 100644 Binary files a/docs/docs/extraction/images/overview-extraction.png and b/docs/docs/extraction/images/overview-extraction.png differ diff --git a/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md b/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md index 1f5f7dd8b..3eb2ea229 100644 --- a/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md +++ b/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md @@ -1,10 +1,5 @@ # Integrate with LangChain, LlamaIndex, and Haystack -!!! note - - This documentation describes NeMo Retriever Library. - - NeMo Retriever Library is commonly used **behind** retrieval-augmented generation (RAG) apps built with popular orchestration frameworks. ## Jupyter examples (LangChain and LlamaIndex) diff --git a/docs/docs/extraction/key-features.md b/docs/docs/extraction/key-features.md index fa1d21ec7..8c0d6c7d7 100644 --- a/docs/docs/extraction/key-features.md +++ b/docs/docs/extraction/key-features.md @@ -1,10 +1,5 @@ # Key features -!!! note - - This documentation describes NeMo Retriever Library. - - NeMo Retriever Library is built for **accuracy**, **throughput**, and **flexible deployment** in enterprise retrieval pipelines. ## Extraction and understanding diff --git a/docs/docs/extraction/multimodal-metadata-schema.md b/docs/docs/extraction/multimodal-metadata-schema.md index fd68ff7af..90a4cd640 100644 --- a/docs/docs/extraction/multimodal-metadata-schema.md +++ b/docs/docs/extraction/multimodal-metadata-schema.md @@ -1,8 +1,3 @@ # Metadata and content schema (multimodal extraction) -!!! note - - This documentation describes NeMo Retriever Library. - - Extracted objects follow the schema and field descriptions in the [Metadata reference](content-metadata.md). Use that page for tables, types, and per-field notes. diff --git a/docs/docs/extraction/nemo-retriever-api-reference.md b/docs/docs/extraction/nemo-retriever-api-reference.md index de30da043..7e4f2c0e0 100644 --- a/docs/docs/extraction/nemo-retriever-api-reference.md +++ b/docs/docs/extraction/nemo-retriever-api-reference.md @@ -1,10 +1,5 @@ # NeMo Retriever API Reference -!!! note - - This documentation describes NeMo Retriever Library. - - ::: nemo_retriever.ingestor ::: nemo_retriever.retriever diff --git a/docs/docs/extraction/nemoretriever-parse.md b/docs/docs/extraction/nemoretriever-parse.md index 5b5da7b8e..23cd5f0d9 100644 --- a/docs/docs/extraction/nemoretriever-parse.md +++ b/docs/docs/extraction/nemoretriever-parse.md @@ -1,10 +1,5 @@ # Advanced Visual Parsing with Nemotron Parse -!!! note - - This documentation describes NeMo Retriever Library. - - For scanned documents, or documents with complex layouts, we recommend that you use [nemotron-parse](https://build.nvidia.com/nvidia/nemotron-parse). Nemotron parse provides higher-accuracy text extraction. diff --git a/docs/docs/extraction/ngc-api-key.md b/docs/docs/extraction/ngc-api-key.md index 9ff911149..8e205aca7 100644 --- a/docs/docs/extraction/ngc-api-key.md +++ b/docs/docs/extraction/ngc-api-key.md @@ -1,10 +1,5 @@ # Generate Your NGC Keys -!!! note - - This documentation describes NeMo Retriever Library. - - NGC contains many public images, models, and datasets that can be pulled immediately without authentication. To push and pull custom images, you must generate a key and authenticate with NGC. diff --git a/docs/docs/extraction/nimclient.md b/docs/docs/extraction/nimclient.md index 59044cc23..a5976f465 100644 --- a/docs/docs/extraction/nimclient.md +++ b/docs/docs/extraction/nimclient.md @@ -3,9 +3,6 @@ The `NimClient` class provides a unified interface for connecting to and interacting with NVIDIA NIM Microservices. This documentation demonstrates how to create custom NIM integrations for use in [NeMo Retriever Library](overview.md) pipelines and User Defined Functions (UDFs). -!!! note - - This documentation describes NeMo Retriever Library. The NimClient architecture consists of two main components: diff --git a/docs/docs/extraction/notebooks.md b/docs/docs/extraction/notebooks.md index e839278d6..90c64a4ff 100644 --- a/docs/docs/extraction/notebooks.md +++ b/docs/docs/extraction/notebooks.md @@ -1,10 +1,5 @@ # Notebooks for NeMo Retriever Library -!!! note - - This documentation describes NeMo Retriever Library. - - To get started using [NeMo Retriever Library](overview.md), you can try one of the ready-made notebooks that are available. ## Dataset Downloads for Benchmarking diff --git a/docs/docs/extraction/nv-ingest-python-api.md b/docs/docs/extraction/nv-ingest-python-api.md index d7167d7ec..7d5df429f 100644 --- a/docs/docs/extraction/nv-ingest-python-api.md +++ b/docs/docs/extraction/nv-ingest-python-api.md @@ -2,9 +2,6 @@ The [NeMo Retriever Library](overview.md) Python API provides a simple and flexible interface for processing and extracting information from various document types, including PDFs. -!!! note - - This documentation describes NeMo Retriever Library. !!! tip diff --git a/docs/docs/extraction/nv-ingest_cli.md b/docs/docs/extraction/nv-ingest_cli.md index 12a28d873..b85a477f8 100644 --- a/docs/docs/extraction/nv-ingest_cli.md +++ b/docs/docs/extraction/nv-ingest_cli.md @@ -1,10 +1,5 @@ # Use the NeMo Retriever Library command line interface -!!! note - - This documentation describes NeMo Retriever Library. - - After you install the Python dependencies, you can use the [NeMo Retriever Library](overview.md) command line interface (CLI). To use the CLI, use the `nemo-retriever` command. diff --git a/docs/docs/extraction/overview.md b/docs/docs/extraction/overview.md index c5c418d7f..3d961da40 100644 --- a/docs/docs/extraction/overview.md +++ b/docs/docs/extraction/overview.md @@ -2,9 +2,6 @@ NVIDIA NeMo Retriever Library (NRL) is a scalable, performance-oriented framework for document content and metadata extraction. It supports both NVIDIA NIM microservices and a wide range of models to find, contextualize, and extract text, tables, charts, and infographics for use in downstream generative and retrieval-augmented applications. -!!! note - - This documentation describes NeMo Retriever Library. NeMo Retriever Library enables parallelization of splitting documents into pages where artifacts are classified (such as text, tables, charts, and infographics), extracted, and further contextualized through optical character recognition (OCR) into a well defined JSON schema. From there, NeMo Retriever Library can optionally manage computation of embeddings for the extracted content, diff --git a/docs/docs/extraction/prerequisites.md b/docs/docs/extraction/prerequisites.md index 6fc319635..35a136a0e 100644 --- a/docs/docs/extraction/prerequisites.md +++ b/docs/docs/extraction/prerequisites.md @@ -2,12 +2,6 @@ Before you begin using [NeMo Retriever Library](overview.md), ensure the following software and hardware prerequisites are met. -!!! note - - This documentation describes NeMo Retriever Library. - - - ## Software Requirements - Linux operating systems (Ubuntu 22.04 or later recommended) diff --git a/docs/docs/extraction/production-checklist.md b/docs/docs/extraction/production-checklist.md index 809d4fc18..7e89500bb 100644 --- a/docs/docs/extraction/production-checklist.md +++ b/docs/docs/extraction/production-checklist.md @@ -1,10 +1,5 @@ # Production checklist -!!! note - - This documentation describes NeMo Retriever Library. - - Use this checklist before you run NeMo Retriever Library in production. Pair it with [Troubleshoot](troubleshoot.md) and your internal security review. **Security and access** diff --git a/docs/docs/extraction/published-metrics-comparisons.md b/docs/docs/extraction/published-metrics-comparisons.md index 2e289ab6e..ff85a0548 100644 --- a/docs/docs/extraction/published-metrics-comparisons.md +++ b/docs/docs/extraction/published-metrics-comparisons.md @@ -1,10 +1,5 @@ # Published metrics and comparisons -!!! note - - This documentation describes NeMo Retriever Library. - - Published scores for parsing and retrieval components (including table-structure metrics such as TEDS where they apply) appear in NVIDIA papers, model cards, and product announcements. Treat published benchmarks as reference points, not guarantees on your own documents. Always [evaluate on your data](evaluate-on-your-data.md). **Starting points** diff --git a/docs/docs/extraction/python-api-reference.md b/docs/docs/extraction/python-api-reference.md index 2911ae3b0..bb21e99f7 100644 --- a/docs/docs/extraction/python-api-reference.md +++ b/docs/docs/extraction/python-api-reference.md @@ -2,9 +2,6 @@ The [NeMo Retriever Library](overview.md) Python API provides a simple and flexible interface for processing and extracting information from various document types, including PDFs. -!!! note - - This documentation describes NeMo Retriever Library. !!! tip diff --git a/docs/docs/extraction/quickstart-guide.md b/docs/docs/extraction/quickstart-guide.md index 73907474c..9b1a3a9b2 100644 --- a/docs/docs/extraction/quickstart-guide.md +++ b/docs/docs/extraction/quickstart-guide.md @@ -1,10 +1,5 @@ # Deploy With Docker Compose (Self-Hosted) for NeMo Retriever Library -!!! note - - This documentation describes NeMo Retriever Library. - - This guide helps you get started using [NeMo Retriever Library](overview.md) in self-hosted mode. @@ -436,17 +431,7 @@ The default [docker-compose.yaml](https://github.com/NVIDIA/NeMo-Retriever/blob/ For RTX Pro 6000 Server Edition and other GPUs with limited VRAM, use the override that best matches your GPU memory (for example, `docker-compose.l40s.yaml` or `docker-compose.a10g.yaml`). -### Example: Using the VLM Profile for Infographic Captioning - -Infographics often combine text, charts, and diagrams into complex visuals. Vision-language model (VLM) captioning generates natural language descriptions that capture this complexity, making the content searchable and more accessible for downstream applications. - -To use VLM captioning for infographics, start NeMo Retriever Library with both the `retrieval` and `vlm` profiles by running the following code. -```shell -docker compose \ - -f docker-compose.yaml \ - --profile retrieval \ - --profile vlm up -``` +For the VLM profile and infographic captioning, see the example under [Profile Information](#profile-information) (add `-f docker-compose..yaml` as needed when using an override file). ### Example with A100 40GB @@ -480,7 +465,7 @@ docker compose \ ## Specify MIG slices for NIM models -When you deploy the pipeline with NIM models on MIG‑enabled GPUs, MIG device slices are requested and scheduled through the `values.yaml` file for the corresponding NIM microservice. For IBM Content-Aware Storage (CAS) deployments, this allows NIM pods to land only on nodes that expose the desired MIG profiles [raw.githubusercontent](https://raw.githubusercontent.com/NVIDIA/NeMo-Retriever/main/helm/README.md). +When you deploy the pipeline with NIM models on MIG‑enabled GPUs, MIG device slices are requested and scheduled through the `values.yaml` file for the corresponding NIM microservice. For IBM Content-Aware Storage (CAS) deployments, this allows NIM pods to land only on nodes that expose the desired MIG profiles (see the [Helm chart README](https://github.com/NVIDIA/NeMo-Retriever/blob/main/helm/README.md)). To target a specific MIG profile—for example, a 3g.20gb slice on an A100, which is a hardware-partitioned virtual GPU instance that gives your workload a fixed mid-sized share of the A100’s compute plus 20 GB of dedicated GPU memory and behaves like a smaller independent GPU—for a given NIM, configure the `resources` and `nodeSelector` under that NIM’s values path in `values.yaml`. diff --git a/docs/docs/extraction/quickstart-library-mode.md b/docs/docs/extraction/quickstart-library-mode.md index ce932e4a1..bda5a4414 100644 --- a/docs/docs/extraction/quickstart-library-mode.md +++ b/docs/docs/extraction/quickstart-library-mode.md @@ -1,9 +1,5 @@ # Deploy Without Containers (Library Mode) for NeMo Retriever Library -!!! note - - This documentation describes NeMo Retriever Library. - Use the [Quick Start for NeMo Retriever Library](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/nemo_retriever/README.md) to set up and run the NeMo Retriever Library locally, so you can build a GPU‑accelerated, multimodal RAG ingestion pipeline that parses PDFs, HTML, text, audio, and video into LanceDB vector embeddings, integrates with Nemotron RAG models (locally or via NIM endpoints), which includes Ray‑based scaling with built‑in recall evaluation. Python 3.12 or later is required (refer to [Prerequisites](prerequisites.md)). ## `run_pipeline` diff --git a/docs/docs/extraction/ray-logging.md b/docs/docs/extraction/ray-logging.md index 9e4add405..3db3b07ef 100644 --- a/docs/docs/extraction/ray-logging.md +++ b/docs/docs/extraction/ray-logging.md @@ -1,10 +1,5 @@ # Configure Ray Logging -!!! note - - This documentation describes NeMo Retriever Library. - - [NeMo Retriever Library](overview.md) uses [Ray](https://docs.ray.io/en/latest/index.html) for logging. You can use environment variables for fine-grained control over [Ray's logging behavior](https://docs.ray.io/en/latest/ray-observability/user-guides/configure-logging.html). In addition, NeMo Retriever Library provides preset configurations that you can use to quickly update Ray logging behavior. diff --git a/docs/docs/extraction/releasenotes.md b/docs/docs/extraction/releasenotes.md index 1f26ac85e..6aad51833 100644 --- a/docs/docs/extraction/releasenotes.md +++ b/docs/docs/extraction/releasenotes.md @@ -2,9 +2,6 @@ This documentation contains the release notes for [NeMo Retriever Library](overview.md). -!!! note - - This documentation describes NeMo Retriever Library. ## 26.03 Release Notes (26.3.0) diff --git a/docs/docs/extraction/reranking.md b/docs/docs/extraction/reranking.md index 65badf87b..83db53894 100644 --- a/docs/docs/extraction/reranking.md +++ b/docs/docs/extraction/reranking.md @@ -1,10 +1,5 @@ # Reranking -!!! note - - This documentation describes NeMo Retriever Library. - - A **reranker** model re-scores the top candidates from initial retrieval so the final top-K better matches the query. NeMo Retriever Library evaluation and deployments can integrate reranker NIMs (refer to the [Support matrix](support-matrix.md) for availability and GPU considerations). **When to enable reranking** diff --git a/docs/docs/extraction/resources-links.md b/docs/docs/extraction/resources-links.md index 7fc8a80a0..17e21372b 100644 --- a/docs/docs/extraction/resources-links.md +++ b/docs/docs/extraction/resources-links.md @@ -1,10 +1,5 @@ # Resources -!!! note - - This documentation describes NeMo Retriever Library. - - External pages for product context, catalogs, and solution blueprints. ## Product and catalog diff --git a/docs/docs/extraction/scaling-modes.md b/docs/docs/extraction/scaling-modes.md index adfba617e..d0026bcf5 100644 --- a/docs/docs/extraction/scaling-modes.md +++ b/docs/docs/extraction/scaling-modes.md @@ -5,11 +5,6 @@ This guide covers how resource scaling modes work across stages in [NeMo Retriev - **Static scaling**: Each pipeline stage runs a fixed number of replicas based on heuristics (memory-aware). Good for consistent latency; higher steady-state memory usage. - **Dynamic scaling**: Only the source stage is fixed; other stages scale up/down based on observed resource pressure. Better memory efficiency; may briefly pause to spin replicas back up after idle periods. -!!! note - - This documentation describes NeMo Retriever Library. - - ## When to choose which diff --git a/docs/docs/extraction/self-host-nims-when-to-use.md b/docs/docs/extraction/self-host-nims-when-to-use.md index 5d078c2b2..6f6a88265 100644 --- a/docs/docs/extraction/self-host-nims-when-to-use.md +++ b/docs/docs/extraction/self-host-nims-when-to-use.md @@ -1,10 +1,5 @@ # When to self-host NIMs -!!! note - - This documentation describes NeMo Retriever Library. - - Self-hosted NIMs run on your GPUs or air-gapped hardware, typically with Kubernetes and the [NIM Operator](https://docs.nvidia.com/nim-operator/latest/index.html). Consider self-hosting when: diff --git a/docs/docs/extraction/semantic-hybrid-retrieval.md b/docs/docs/extraction/semantic-hybrid-retrieval.md index eae94123a..a02d9f082 100644 --- a/docs/docs/extraction/semantic-hybrid-retrieval.md +++ b/docs/docs/extraction/semantic-hybrid-retrieval.md @@ -1,10 +1,5 @@ # Semantic and hybrid retrieval -!!! note - - This documentation describes NeMo Retriever Library. - - **Semantic retrieval** uses dense embeddings to find content that is similar in meaning to a query. **Hybrid retrieval** combines dense vectors with sparse or lexical signals (for example, BM25-style full-text) and fuses ranked lists for better recall on keyword-heavy queries. In NeMo Retriever Library, use these resources: diff --git a/docs/docs/extraction/support-matrix.md b/docs/docs/extraction/support-matrix.md index 22ef162b9..90876957b 100644 --- a/docs/docs/extraction/support-matrix.md +++ b/docs/docs/extraction/support-matrix.md @@ -2,10 +2,6 @@ Before you begin using [NeMo Retriever Library](overview.md), ensure that you have the hardware for your use case. -!!! note - - This documentation describes NeMo Retriever Library. - ## Core and Advanced Pipeline Features diff --git a/docs/docs/extraction/supported-file-types.md b/docs/docs/extraction/supported-file-types.md index 3a9dde0f8..8eaf87b9a 100644 --- a/docs/docs/extraction/supported-file-types.md +++ b/docs/docs/extraction/supported-file-types.md @@ -1,10 +1,5 @@ # Supported file types and formats -!!! note - - This documentation describes NeMo Retriever Library. - - NeMo Retriever Library accepts multiple document and media types. A current list (including PDF, Office formats, HTML, images, audio, and video, some early access) appears in [What is NeMo Retriever Library?](overview.md) under supported file types. **Related** diff --git a/docs/docs/extraction/telemetry.md b/docs/docs/extraction/telemetry.md index 8e2f75063..645ac9c11 100644 --- a/docs/docs/extraction/telemetry.md +++ b/docs/docs/extraction/telemetry.md @@ -2,10 +2,6 @@ You can view telemetry data for [NeMo Retriever Library](overview.md). -!!! note - - This documentation describes NeMo Retriever Library. - ## OpenTelemetry diff --git a/docs/docs/extraction/text-layout-extraction.md b/docs/docs/extraction/text-layout-extraction.md index 173b46521..3d3cb0926 100644 --- a/docs/docs/extraction/text-layout-extraction.md +++ b/docs/docs/extraction/text-layout-extraction.md @@ -1,10 +1,5 @@ # Text and layout extraction -!!! note - - This documentation describes NeMo Retriever Library. - - For PDFs, NeMo Retriever Library typically uses **pdfium**-based extraction with configurable depth and paths. Scanned or mixed pages may use hybrid or OCR-oriented methods. For `extract_method` options such as `pdfium`, `pdfium_hybrid`, and `ocr`, refer to the [Python API reference](python-api-reference.md). **Related** diff --git a/docs/docs/extraction/throughput-is-dataset-dependent.md b/docs/docs/extraction/throughput-is-dataset-dependent.md index dc3349e92..399d116c8 100644 --- a/docs/docs/extraction/throughput-is-dataset-dependent.md +++ b/docs/docs/extraction/throughput-is-dataset-dependent.md @@ -1,10 +1,5 @@ # Why Throughput Is Dataset-Dependent -!!! note - - This documentation describes NeMo Retriever Library. - - A single headline metric can drastically misrepresent system efficiency. The amount of compute that you need to process a dataset depends far more on its content and how your pipeline operates than on its disk size. This documentation explains why, and offers you better ways to measure and report throughput. diff --git a/docs/docs/extraction/troubleshoot.md b/docs/docs/extraction/troubleshoot.md index 9b65100bb..7304bd16d 100644 --- a/docs/docs/extraction/troubleshoot.md +++ b/docs/docs/extraction/troubleshoot.md @@ -2,10 +2,6 @@ Use this documentation to troubleshoot issues that arise when you use [NeMo Retriever Library](overview.md). -!!! note - - This documentation describes NeMo Retriever Library. - ## Can't process long, non-language text strings diff --git a/docs/docs/extraction/user-defined-functions.md b/docs/docs/extraction/user-defined-functions.md index fb6e5061a..ed105d275 100644 --- a/docs/docs/extraction/user-defined-functions.md +++ b/docs/docs/extraction/user-defined-functions.md @@ -3,10 +3,6 @@ User-Defined Functions (UDFs) allow you to inject custom processing logic into the [NeMo Retriever Library](overview.md) pipeline at specific stages. This guide covers how to write, validate, and submit UDFs using both the CLI and the Python client interface. -!!! note - - This documentation describes NeMo Retriever Library. - ## Quickstart diff --git a/docs/docs/extraction/user-defined-stages.md b/docs/docs/extraction/user-defined-stages.md index 4b041f5a3..bac451386 100644 --- a/docs/docs/extraction/user-defined-stages.md +++ b/docs/docs/extraction/user-defined-stages.md @@ -6,10 +6,6 @@ By following these steps, your Lambda stages are robust, signature-validated, plug-and-play for your RayPipeline, and operate on a well-defined DataFrame payload and metadata structure. -!!! note - - This documentation describes NeMo Retriever Library. - To add user-defined stages to your pipeline, you need the following: diff --git a/docs/docs/extraction/v2-api-guide.md b/docs/docs/extraction/v2-api-guide.md index 3b7b25643..5c45adb13 100644 --- a/docs/docs/extraction/v2-api-guide.md +++ b/docs/docs/extraction/v2-api-guide.md @@ -1,10 +1,5 @@ # NeMo Retriever Library V2 API Guide: PDF Pre Splitting -!!! note - - This documentation describes NeMo Retriever Library. - - > **TL;DR:** V2 API automatically splits large PDFs into chunks for faster parallel processing. > > **Python:** Enable with `message_client_kwargs={"api_version": "v2"}` and configure chunk size with `.pdf_split_config(pages_per_chunk=64)`. diff --git a/docs/docs/extraction/vector-db-partners.md b/docs/docs/extraction/vector-db-partners.md index 1b7899fa8..057122ab8 100644 --- a/docs/docs/extraction/vector-db-partners.md +++ b/docs/docs/extraction/vector-db-partners.md @@ -1,10 +1,5 @@ # Vector database partners -!!! note - - This documentation describes NeMo Retriever Library. - - NeMo Retriever Library integrates with vector databases used for RAG collections. Documentation here focuses on stores used in the library and harnesses, such as LanceDB and Milvus, and cuVS where it applies. Refer to [Vector databases](data-store.md) and [Chunking and splitting](chunking.md). **Related** diff --git a/docs/docs/extraction/vlm-embed.md b/docs/docs/extraction/vlm-embed.md index 941b727ad..fdce43257 100644 --- a/docs/docs/extraction/vlm-embed.md +++ b/docs/docs/extraction/vlm-embed.md @@ -8,10 +8,6 @@ The model can embed documents in the form of an image, text, or a combination of Documents can then be retrieved given a user query in text form. The model supports images that contain text, tables, charts, and infographics. -!!! note - - This documentation describes NeMo Retriever Library. - ## Configure and Run the Multimodal NIM diff --git a/docs/docs/extraction/workflow-agentic-retrieval.md b/docs/docs/extraction/workflow-agentic-retrieval.md index b7aa51021..63ccc1040 100644 --- a/docs/docs/extraction/workflow-agentic-retrieval.md +++ b/docs/docs/extraction/workflow-agentic-retrieval.md @@ -1,10 +1,5 @@ # Workflow: Agentic retrieval -!!! note - - This documentation describes NeMo Retriever Library. - - **Agentic retrieval** describes patterns where a planner or tool-using agent queries retrieval systems in a loop (often combining multiple searches, filters, and rerankers) instead of sending a single static query. NeMo Retriever Library provides ingestion, embedding, storage, and retrieval building blocks (jobs, chunking, vector stores, hybrid search, reranking) that you orchestrate in application code or frameworks. diff --git a/docs/docs/extraction/workflow-build-searchable-collection.md b/docs/docs/extraction/workflow-build-searchable-collection.md index 0bb409e87..9f67ac1ca 100644 --- a/docs/docs/extraction/workflow-build-searchable-collection.md +++ b/docs/docs/extraction/workflow-build-searchable-collection.md @@ -1,10 +1,5 @@ # Workflow: Build a searchable collection -!!! note - - This documentation describes NeMo Retriever Library. - - After [document ingestion](workflow-document-ingestion.md), configure [chunking](chunking.md) and your [vector database](data-store.md) so extracted content is embedded, indexed, and ready for search. Technical detail for storage and chunking is in the **Embedding, indexing, and storage** section of the navigation ([Vector databases](data-store.md), [Chunking and splitting](chunking.md)). diff --git a/docs/docs/extraction/workflow-document-ingestion.md b/docs/docs/extraction/workflow-document-ingestion.md index 82f2fd941..e7cff6ff4 100644 --- a/docs/docs/extraction/workflow-document-ingestion.md +++ b/docs/docs/extraction/workflow-document-ingestion.md @@ -1,10 +1,5 @@ # Workflow: Document ingestion -!!! note - - This documentation describes NeMo Retriever Library. - - **Document ingestion** is the step where NeMo Retriever Library reads your files (PDFs, Office documents, images, and other [supported formats](supported-file-types.md)), runs extraction and optional enrichment, and returns structured content you can chunk, embed, and index. Follow these steps: diff --git a/docs/docs/extraction/workflow-e2e-blueprints.md b/docs/docs/extraction/workflow-e2e-blueprints.md index aa4d6c0f0..9935e7f66 100644 --- a/docs/docs/extraction/workflow-e2e-blueprints.md +++ b/docs/docs/extraction/workflow-e2e-blueprints.md @@ -1,8 +1,3 @@ # Workflow: End-to-end RAG with NVIDIA Blueprints -!!! note - - This documentation describes NeMo Retriever Library. - - For NVIDIA AI Blueprint links, solution cards, enterprise RAG resources, and related product landing pages, refer to [NVIDIA AI Blueprints and product links](resources-links.md). diff --git a/docs/docs/extraction/workflow-query-rerank.md b/docs/docs/extraction/workflow-query-rerank.md index 9f216c30f..7c4a2620b 100644 --- a/docs/docs/extraction/workflow-query-rerank.md +++ b/docs/docs/extraction/workflow-query-rerank.md @@ -1,10 +1,5 @@ # Workflow: Query and rerank -!!! note - - This documentation describes NeMo Retriever Library. - - After documents are [ingested](workflow-document-ingestion.md), [chunked](chunking.md), and [indexed](data-store.md), applications run retrieval (semantic and optionally hybrid search) and optional reranking to improve top-K quality. Follow these steps: diff --git a/docs/docs/extraction/workflow-video-ocr.md b/docs/docs/extraction/workflow-video-ocr.md index 583e9f7bd..92c69e8e9 100644 --- a/docs/docs/extraction/workflow-video-ocr.md +++ b/docs/docs/extraction/workflow-video-ocr.md @@ -1,10 +1,5 @@ # Workflow: Video processing with OCR -!!! note - - This documentation describes NeMo Retriever Library. - - For video assets, NeMo Retriever Library can combine audio or speech processing with visual text extraction when OCR applies to frames or derived images. For the audio and speech path, refer to [Speech and audio](audio.md) for RIVA ASR and related ingestion paths.