Add lost info on multimodal config: indexingFragments and searchFragments

[CaroFG]

Before the API ref revamp, the embedder settings contained info on how to configure a multimodal embedder. Today, we only have the https://www.meilisearch.com/docs/learn/ai_powered_search/image_search_with_multimodal_embeddings guide, which is very useful, but it lacks some important info like:

What params are required, which ones are optional. What are the objects required by Meili and which ones depend on the chosen model.

[CaroFG]

indexingFragments

This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it:

curl \
  -X PATCH 'MEILISEARCH_URL/experimental-features/' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "multimodal": true
  }'

indexingFragments specifies which fields in your documents should be used to generate multimodal embeddings. It must be an object with the following structure:

  "FRAGMENT_NAME": {
    "value": {
      …
    }
  }

FRAGMENT_NAME can be any valid string. It must contain a single field, value. value must then follow your chosen model's specifications.

For example, for VoyageAI's multimodal embedding route, value must be an object containing a content field. content itself must contain an array of objects with a type field. Depending on type's value, you must include either text, image_url, or image_base64:

{
  "VOYAGE_FRAGMENT_NAME_A": {
    "value": {
      "content": [
        {
          "type": "text",
          "text": "A document called {{doc.title}} that can be described as {{doc.description}}"
        }
      ]
    }
  },
  "VOYAGE_FRAGMENT_NAME_B": {
    "value": {
      "content": [
        {
          "type": "image_url",
          "image_url": "{{doc.image_url}}"
        }
      ]
    }
  },
}

Use Liquid templates to interpolate document data into the fragment fields, where doc gives you access to all fields within a document.

If a Liquid template appearing inside of a fragment cannot be rendered, no embedding will be generated for that fragment and that document. If a document has no indexing fragments, it will not be returned in multimodal searches. In most cases, a fragment is not rendered because a field it references is missing in the document.

This is different from embeddings based on documentTemplate, which abort the indexing task if the document template cannot be rendered for a document.

You can check which documents have embeddings for a given fragment using vector filters.

indexingFragments is optional when using the rest source.

indexingFragments is incompatible with all other embedder sources.

Specifying a documentTemplate in an embedder using indexingFragments will result in an error.

You must specify at least one valid fragment in searchFragments when using indexingFragments.

searchFragments

This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it:

curl \
  -X PATCH 'MEILISEARCH_URL/experimental-features/' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "multimodal": true
  }'

searchFragments instructs Meilisearch how to parse fields present in a query's media search parameter. It must be an object following the same structure as the indexingFragments object:

  "FRAGMENT_NAME": {
    "value": {
      …
    }
  }

As with indexingFragments, the content of value should follow your model's specification.

Use Liquid templates to interpolate search query data into the fragment fields, where {{media.*}} gives you access to all multimodal data received with a query and {{q}} gives you access to the regular textual query:

{
  "SEARCH_FRAGMENT_A": {
    "value": {
      "content": [
        {
          "type": "image_base64",
          "image_base64": "data:{{media.image.mime}};base64,{{media.image.data}}"
        }
      ]
    }
  },
  "SEARCH_FRAGMENT_B": {
    "value": {
      "content": [
        {
          "type": "text",
          "text": "{{q}}"
        }
      ]
    }
  }
}

searchFragments is optional when using the rest source.

searchFragments is incompatible with all other embedder sources.

You must specify at least one valid fragment in indexingFragments when using searchFragments.