diff --git a/.gitignore b/.gitignore index a45da7cdd7d..70f1b184fb2 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,11 @@ node_modules cdk* .dev* CLAUDE.md +mcp.json + +# Copilot customization files - workspace agent and instructions +.github/copilot-instructions.md +.github/agents/*.agent.md + +# VS Code Copilot cache +.vscode/.copilot-cache/ diff --git a/_analyzers/token-filters/hunspell.md b/_analyzers/token-filters/hunspell.md index 0e8395cae8d..84dc65da58d 100644 --- a/_analyzers/token-filters/hunspell.md +++ b/_analyzers/token-filters/hunspell.md @@ -28,7 +28,7 @@ Parameter | Required/Optional | Data type | Description `dedup` | Optional | Boolean | Determines whether to remove multiple duplicate stemming terms for the same token. Default is `true`. `dictionary` | Optional | Array of strings | Configures the dictionary files to be used for the Hunspell dictionary. Default is all files in the `/hunspell/` directory if `ref_path` is not specified or all files in the `//hunspell//` directory when `ref_path` is specified. See [Custom dictionary loading with ref_path](#custom-dictionary-loading). `longest_only` | Optional | Boolean | Specifies whether only the longest stemmed version of the token should be returned. Default is `false`. -`updateable` | Optional | Boolean | When set to `true`, the filter operates in search-time analysis mode, allowing dictionaries to be hot-reloaded by using the [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/) API without restarting the node. Default is `false`. **Introduced 3.7.** +`updateable` | Optional | Boolean | When set to `true`, the filter operates in search-time analysis mode, allowing dictionaries to be hot-reloaded by using the [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/index-other/refresh-analyzer/) API without restarting the node. Default is `false`. **Introduced 3.7.** ## Example @@ -197,7 +197,7 @@ To hot-reload a Hunspell dictionary, follow these steps: 1. Replace the `.aff` and `.dic` files on disk on every node that holds a shard for the index. -1. Call the [Refresh Search Analyzer API]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/). When `reload_cached_resources` is `false` (the default), the API rebuilds analyzer factories but reuses the previously cached Hunspell dictionary. Specify `reload_cached_resources=true` to force the dictionary to be reloaded from disk: +1. Call the [Refresh Search Analyzer API]({{site.url}}{{site.baseurl}}/im-plugin/index-other/refresh-analyzer/). When `reload_cached_resources` is `false` (the default), the API rebuilds analyzer factories but reuses the previously cached Hunspell dictionary. Specify `reload_cached_resources=true` to force the dictionary to be reloaded from disk: ```json POST /_plugins/_refresh_search_analyzers/my_index?reload_cached_resources=true diff --git a/_api-reference/alias/alias-exists.md b/_api-reference/alias/alias-exists.md index 871272ed474..5962aec1f8e 100644 --- a/_api-reference/alias/alias-exists.md +++ b/_api-reference/alias/alias-exists.md @@ -77,4 +77,4 @@ If you use the Security plugin, make sure you have the appropriate permissions: ## Related documentation -For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). \ No newline at end of file +For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). \ No newline at end of file diff --git a/_api-reference/alias/aliases-api.md b/_api-reference/alias/aliases-api.md index b9d9bf9e12d..5ebfb842774 100644 --- a/_api-reference/alias/aliases-api.md +++ b/_api-reference/alias/aliases-api.md @@ -16,7 +16,7 @@ The Manage aliases API performs multiple index alias operations in a single atom This API is distinct from the [Create or update alias API]({{site.url}}{{site.baseurl}}/api-reference/alias/create-alias/), which operates on a single alias at a time and uses different request parameters. Use the Manage aliases API for bulk operations and atomic transactions involving multiple aliases or indexes. -For conceptual information about index aliases, including use cases and examples, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). +For conceptual information about index aliases, including use cases and examples, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). ## Endpoints @@ -521,4 +521,4 @@ If you use the Security plugin, make sure you have the appropriate permissions: ## Related documentation -For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). \ No newline at end of file +For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). \ No newline at end of file diff --git a/_api-reference/alias/create-alias.md b/_api-reference/alias/create-alias.md index e5bc8407e3e..ca6cf8da043 100644 --- a/_api-reference/alias/create-alias.md +++ b/_api-reference/alias/create-alias.md @@ -12,7 +12,7 @@ redirect_from: **Introduced 1.0** {: .label .label-purple } -The Create or Update Alias API adds one or more indexes to an index alias or updates the settings for an existing alias. For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). +The Create or Update Alias API adds one or more indexes to an index alias or updates the settings for an existing alias. For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). The Create or Update Alias API is distinct from the [Manage aliases API]({{site.url}}{{site.baseurl}}/api-reference/alias/aliases-api/), which supports the addition and removal of aliases and the removal of indexes with their aliases. In contrast, the following API only supports adding or updating an alias without updating the index itself. Each API also uses different request body parameters. {: .note} @@ -324,4 +324,4 @@ If you use the Security plugin, make sure you have the appropriate permissions: ## Related documentation -For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). +For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). diff --git a/_api-reference/alias/delete-alias.md b/_api-reference/alias/delete-alias.md index edf641d14f2..6ee2d16c0ee 100644 --- a/_api-reference/alias/delete-alias.md +++ b/_api-reference/alias/delete-alias.md @@ -86,4 +86,4 @@ If you use the Security plugin, make sure you have the appropriate permissions: ## Related documentation -For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). \ No newline at end of file +For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). \ No newline at end of file diff --git a/_api-reference/alias/get-alias.md b/_api-reference/alias/get-alias.md index 36410eaf72b..7f6885a24b7 100644 --- a/_api-reference/alias/get-alias.md +++ b/_api-reference/alias/get-alias.md @@ -223,4 +223,4 @@ If you use the Security plugin, make sure you have the appropriate permissions: ## Related documentation -For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). \ No newline at end of file +For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). \ No newline at end of file diff --git a/_api-reference/alias/index.md b/_api-reference/alias/index.md index bf4f9e183b4..3e327e47339 100644 --- a/_api-reference/alias/index.md +++ b/_api-reference/alias/index.md @@ -15,7 +15,7 @@ redirect_from: OpenSearch index alias APIs allow you to create, update, delete, and retrieve information about aliases. -For conceptual information about index aliases, including use cases and examples, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). +For conceptual information about index aliases, including use cases and examples, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). ## Available APIs @@ -32,4 +32,4 @@ OpenSearch supports the following index alias APIs. ## Related documentation -For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). \ No newline at end of file +For more information about index aliases, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). \ No newline at end of file diff --git a/_api-reference/document-apis/reindex.md b/_api-reference/document-apis/reindex.md index 627225ab7ca..5f3e031bec4 100644 --- a/_api-reference/document-apis/reindex.md +++ b/_api-reference/document-apis/reindex.md @@ -28,7 +28,7 @@ Before using the Reindex API, note the following requirements and limitations: Reindexing large datasets can be resource intensive and may impact cluster performance. Monitor cluster health during reindex operations and consider using throttling parameters for production environments. For more information, see [Performance optimization](#performance-optimization). {: .warning } -For a practical, tutorial-style guide to reindexing with common use cases and examples, see [Reindex data]({{site.url}}{{site.baseurl}}/im-plugin/reindex-data/). +For a practical, tutorial-style guide to reindexing with common use cases and examples, see [Reindex data]({{site.url}}{{site.baseurl}}/im-plugin/reindex/index/). {: .tip } Unlike update operations that modify documents within the same index, reindex operations work on different sources and destinations. Thus, version conflicts are unlikely. The `version_type` parameter controls how OpenSearch handles document versions during reindexing. By default, version conflicts stop the reindex process. To continue reindexing when conflicts occur, set the `conflicts` parameter to `proceed`. The response will include a count of version conflicts encountered. Other error types are unaffected by the `conflicts` parameter. diff --git a/_api-reference/index-apis/clone.md b/_api-reference/index-apis/clone.md index 9aea7ab8343..13ea276afdf 100644 --- a/_api-reference/index-apis/clone.md +++ b/_api-reference/index-apis/clone.md @@ -136,7 +136,7 @@ Field | Data type | Description `settings` | Object | Configuration options for the target index. For a list of index settings, see [Index settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index-settings/). Optional. `settings.index.number_of_shards` | Integer | The number of primary shards in the target index. This value must equal the number of primary shards in the source index. Optional. Default is the same as the source index. `settings.index.number_of_replicas` | Integer | The number of replica shards for each primary shard in the target index. Optional. Default is the same as the source index. -`aliases` | Object | Index aliases to apply to the target index. Each key is an alias name, and the value is an alias configuration object. For more information, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). Optional. +`aliases` | Object | Index aliases to apply to the target index. Each key is an alias name, and the value is an alias configuration object. For more information, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). Optional. **Note**: You cannot specify mappings in the clone request. The mappings from the source index are automatically used for the target index. {: .note} diff --git a/_api-reference/index-apis/create-index-template.md b/_api-reference/index-apis/create-index-template.md index 6128ef812ba..32174543f21 100644 --- a/_api-reference/index-apis/create-index-template.md +++ b/_api-reference/index-apis/create-index-template.md @@ -48,7 +48,7 @@ Parameter | Type | Description `priority` | Integer | A number that determines which index templates take precedence during the creation of a new index or data stream. OpenSearch chooses the template with the highest priority. When no priority is given, the template is assigned a `0`, signifying the lowest priority. Optional. `template` | Object | The template that includes the `aliases`, `mappings`, or `settings` for the index. For more information, see [#template]. Optional. `version` | Integer | The version number used to manage index templates. Version numbers are not automatically set by OpenSearch. Optional. -`context` | Object | (Experimental) The `context` parameter provides use-case-specific predefined templates that can be applied to an index. Among all settings and mappings declared for a template, context templates hold the highest priority. For more information, see [index-context]({{site.url}}{{site.baseurl}}/im-plugin/index-context/). +`context` | Object | (Experimental) The `context` parameter provides use-case-specific predefined templates that can be applied to an index. Among all settings and mappings declared for a template, context templates hold the highest priority. For more information, see [index-context]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-context). ### Template diff --git a/_api-reference/index-apis/create-index.md b/_api-reference/index-apis/create-index.md index 3d04460f059..2a9b89a604d 100644 --- a/_api-reference/index-apis/create-index.md +++ b/_api-reference/index-apis/create-index.md @@ -78,7 +78,7 @@ Field | Data type | Description `settings.number_of_replicas` | Integer | Simplified syntax for specifying the number of replica shards without the `index` prefix. Optional. `mappings` | Object | Field mappings for documents in the index. Defines the data type and properties for each field. For more information, see [Mappings]({{site.url}}{{site.baseurl}}/field-types/). Optional. `mappings.properties` | Object | Defines the fields and their data types in documents. Each key is a field name, and each value is a field definition object. Optional. -`aliases` | Object | Index aliases for the index. Each key is an alias name, and each value is an alias definition object. For more information, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). Optional. +`aliases` | Object | Index aliases for the index. Each key is an alias name, and each value is an alias definition object. For more information, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). Optional. **Note**: You do not have to explicitly specify the `index` section inside the `settings` section. You can use the simplified syntax instead. {: .note} diff --git a/_api-reference/index-apis/get-index.md b/_api-reference/index-apis/get-index.md index f3300300875..aaca5e5c2df 100644 --- a/_api-reference/index-apis/get-index.md +++ b/_api-reference/index-apis/get-index.md @@ -126,7 +126,7 @@ The response contains a separate object for each index, for which the key is the Field | Data type | Description :--- | :--- | :--- -`aliases` | Object | Index aliases associated with the index. Each key is an alias name, and each value is an alias configuration object. For more information, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/index-alias/). +`aliases` | Object | Index aliases associated with the index. Each key is an alias name, and each value is an alias configuration object. For more information, see [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). `mappings` | Object | Field mappings for documents in the index. Defines the data type and properties for each field. For more information, see [Mappings]({{site.url}}{{site.baseurl}}/field-types/). `settings` | Object | Index settings that control index behavior, such as the number of shards and replicas. For more information, see [Index settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index-settings/). diff --git a/_api-reference/index-apis/index-operations.md b/_api-reference/index-apis/index-operation-apis.md similarity index 94% rename from _api-reference/index-apis/index-operations.md rename to _api-reference/index-apis/index-operation-apis.md index c90a9773e7f..2ed331c6adb 100644 --- a/_api-reference/index-apis/index-operations.md +++ b/_api-reference/index-apis/index-operation-apis.md @@ -1,19 +1,17 @@ --- layout: default -title: Index operations +title: Index operations APIs parent: Index APIs nav_order: 30 has_children: true has_toc: false --- -# Index operations +# Index operation APIs Index operations APIs provide advanced functionality for maintaining and optimizing indexes in your OpenSearch cluster. These operations help you manage index performance, data organization, and cluster efficiency. -## Available APIs - -OpenSearch supports the following index operations APIs. +OpenSearch supports the following index operation APIs: | API | Description | |-----|-------------| diff --git a/_api-reference/index-apis/index.md b/_api-reference/index-apis/index.md index 3f387b0f669..ad2398bc886 100644 --- a/_api-reference/index-apis/index.md +++ b/_api-reference/index-apis/index.md @@ -22,11 +22,11 @@ The following index API operations are available, organized by category: - [Alias APIs]({{site.url}}{{site.baseurl}}/api-reference/alias/) - Create, update, delete, and retrieve information about index aliases - [Core index APIs]({{site.url}}{{site.baseurl}}/api-reference/index-apis/core-index-apis/) - Fundamental operations for managing the lifecycle of indexes -- [Index operations]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index-operations/) - Advanced functionality for maintaining and optimizing indexes +- [Index operations]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index-operation-apis/) - Advanced functionality for maintaining and optimizing indexes - [Index settings and mappings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index-settings-mappings/) - Configure and modify index behavior and structure - [Index templates]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index-templates/) - Create and manage templates for automatic index configuration - [Index blocks and allocation]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index-blocks-allocation/) - Control index access restrictions and shard allocation -- [Data stream stats]({{site.url}}{{site.baseurl}}/api-reference/index-apis/data-stream-stats/) - Returns statistics for data streams +- [Data streams]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/) - Create, delete, and get information and statistics about data streams - [Dangling index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/dangling-index/) - Manage indexes that exist on disk but are not part of cluster state If you use the Security plugin, make sure you have the appropriate permissions. diff --git a/_api-reference/index-apis/modify-data-stream.md b/_api-reference/index-apis/modify-data-stream.md index b259dd8f706..f27096ac376 100644 --- a/_api-reference/index-apis/modify-data-stream.md +++ b/_api-reference/index-apis/modify-data-stream.md @@ -98,4 +98,4 @@ If you use the Security plugin, make sure you have the appropriate permissions: ## Related documentation - [Data streams]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/) -- [Data Stream Stats API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/data-stream-stats/) +- [Viewing data stream statistics]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#viewing-data-stream-statistics) diff --git a/_api-reference/index-apis/shrink-index.md b/_api-reference/index-apis/shrink-index.md index bf94f43a108..e80c453b3dd 100644 --- a/_api-reference/index-apis/shrink-index.md +++ b/_api-reference/index-apis/shrink-index.md @@ -133,7 +133,7 @@ You can track the progress of shard recovery using the [CAT recovery API]({{site ## Index codec considerations -For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#splits-and-shrinks). +For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/#splits-and-shrinks). ## Example: Shrinking an index diff --git a/_api-reference/index-apis/split.md b/_api-reference/index-apis/split.md index 284d7a6f4f9..11cc5edcc9f 100644 --- a/_api-reference/index-apis/split.md +++ b/_api-reference/index-apis/split.md @@ -121,7 +121,7 @@ The following table lists the available request body fields. All fields are opti ## Index codec considerations -For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#splits-and-shrinks). +For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/#splits-and-shrinks). ## Example: Splitting an index diff --git a/_api-reference/index.md b/_api-reference/index.md index 41e8573b966..c35f61413ee 100644 --- a/_api-reference/index.md +++ b/_api-reference/index.md @@ -48,8 +48,8 @@ OpenSearch supports the following REST APIs: - [Count]({{site.url}}{{site.baseurl}}/api-reference/count/) - [Cross-cluster replication API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/replication-plugin/api/) - [Explain]({{site.url}}{{site.baseurl}}/api-reference/explain/) -- [Index rollups API]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/rollup-api/) -- [Index state management API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/) +- [Index rollups API]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/api-rollup/) +- [Index state management API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/) - [ISM error prevention API]({{site.url}}{{site.baseurl}}/im-plugin/ism/error-prevention/api/) - [Job Scheduler APIs]({{site.url}}{{site.baseurl}}/monitoring-your-cluster/job-scheduler/index/#job-scheduler-apis) - [Vector search API]({{site.url}}{{site.baseurl}}/vector-search/api/) @@ -59,7 +59,7 @@ OpenSearch supports the following REST APIs: - [Performance Analyzer API]({{site.url}}{{site.baseurl}}/monitoring-your-cluster/pa/api/) - [Point in Time API]({{site.url}}{{site.baseurl}}/search-plugins/point-in-time-api/) - [Ranking evaluation]({{site.url}}{{site.baseurl}}/api-reference/rank-eval/) -- [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/) +- [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/index-other/refresh-analyzer/) - [Remote cluster information]({{site.url}}{{site.baseurl}}/api-reference/remote-info/) - [Root cause analysis API]({{site.url}}{{site.baseurl}}/monitoring-your-cluster/pa/rca/api/) - [Snapshot management API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/snapshots/sm-api/) @@ -69,7 +69,7 @@ OpenSearch supports the following REST APIs: - [Security Analytics APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/index/) - [Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/stats-api/) - [Tasks]({{site.url}}{{site.baseurl}}/api-reference/tasks/) -- [Transforms API]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms/transforms-apis/) +- [Transforms API]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms/api-transform/) - [Hot reload TLS certificates]({{site.url}}{{site.baseurl}}/security/configuration/tls/#hot-reloading-tls-certificates) ## gRPC APIs diff --git a/_benchmark/reference/telemetry.md b/_benchmark/reference/telemetry.md index 05874523f33..2907e7e9409 100644 --- a/_benchmark/reference/telemetry.md +++ b/_benchmark/reference/telemetry.md @@ -116,7 +116,7 @@ This device supports the `shard-stats-sample-interval` parameter, which defines ## data-stream-stats -The `data-stream-stats` telemetry device regularly calls the [Data Stream Stats API]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/#step-2-create-a-data-stream) and records one metrics document for cluster-level stats (`_all`) and one metrics document per data stream. +The `data-stream-stats` telemetry device regularly calls the [data stream stats API]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#viewing-data-stream-statistics) and records one metrics document for cluster-level stats (`_all`) and one metrics document per data stream. The following is an example of recorded documents given two data streams in the cluster: diff --git a/_dashboards/im-dashboards/component-templates.md b/_dashboards/im-dashboards/component-templates.md deleted file mode 100644 index 026639fe640..00000000000 --- a/_dashboards/im-dashboards/component-templates.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -layout: default -title: Component templates -parent: Index management -nav_order: 50 ---- - -# Component templates -Introduced 2.7 -{: .label .label-purple } - -Component templates allow you to create a single index pattern that matches multiple indexes. This pattern can include wildcards or regular expressions, enabling you to apply the same setting or mapping to multiple indexes simultaneously. - -Using them with [index templates]({{site.url}}{{site.baseurl}}/im-plugin/index-templates/) can provide a powerful tool for managing large volumes of data. You can create an index template that defines the basic structure and settings of your indexes and then use the component templates to apply the settings to all indexes that match a specific pattern or set of criteria. - -You can create component templates using the Index Management UI. The UI maximizes ease of use for common indexing and data stream administrative operations such as create, read, update, delete (CRUD) and mapping indexes; CRUD and mapping aliases; reindexing; and open/close, shrink, and split indexes, along with the monitoring of actions and logging of audit records. - -The following GIF demonstrates creating a component template. - -![Component template demo]({{site.url}}{{site.baseurl}}/images/admin-ui-index/component.gif) - -## Prerequisites - -This tutorial is intended for admin users who [manage OpenSearch clusters]({{site.url}}{{site.baseurl}}/tuning-your-cluster/cluster/) and are familiar with [index management in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/). - -## Key terms - -It's helpful to understand the following terms before starting this tutorial: - -- *Component template* refers to a reusable building block with settings, mappings, and aliases that can be attached to an index template. -- *Index template* refers to a predefined structure used to organize and store data in a database or search index. - -## Creating component templates using the index management UI - -You can use predefined OpenSearch Dashboards component templates or customize your own, either by creating original templates or by modifying existing templates. Predefined component templates include preconfigured charts, tables, and graphs and are a good starting point for users who are new to OpenSearch Dashboards. Alternatively, customized template components provide you with options for tailoring reports and visualizations that meet your specific requirements and preferences. - -To create template components using the UI, follow these steps: - -1. On the OpenSearch Dashboards main page, select **Index Management** in the navigation menu. -1. In the Index Management window, select **Templates** > **Component templates**. -1. Select **Create** and then define the component template settings. -1. To configure aliases, settings, and mappings, toggle **Use configuration**, as shown in the following image. - - ![Component template use configuration]({{site.url}}{{site.baseurl}}/images/admin-ui-index/component_use_config.png) - -1. Enter details in the aliases, settings, and mappings fields. -1. Select **Create component template**. - -When you create component templates, those templates apply only to new index templates that you create and not to existing index templates. -{: .note } - -## Associating component templates with index templates - -To associate a component template with an index template, follow these steps: - -1. In the **Index Management** navigation menu, select **Templates**. -1. In the Templates window, select **Create template**. -1. Select **Component template** as the method for defining your template. -1. In the **Component template** pane, select **Associate component template**, as shown in the following image. - - ![Component template associate configuration]({{site.url}}{{site.baseurl}}/images/admin-ui-index/associate_component.png) - -1. In the **Associate component template** pop-up window, select the component templates that you want to associate with your index template. -1. Select **Associate**. -1. Select **Preview template** to view the template settings. -1. Select **Create template**. \ No newline at end of file diff --git a/_dashboards/im-dashboards/forcemerge.md b/_dashboards/im-dashboards/forcemerge.md deleted file mode 100644 index 8554664299b..00000000000 --- a/_dashboards/im-dashboards/forcemerge.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -layout: default -title: Force merge -parent: Index management -nav_order: 30 -redirect_from: - - /dashboards/admin-ui-index/forcemerge/ ---- - -# Force merge -Introduced 2.6 -{: .label .label-purple } - -OpenSearch Dashboards allows you to perform a [force merge]({{site.url}}{{site.baseurl}}/im-plugin/ism/error-prevention/index/#force-merge) operation on two or more indexes with **Index Management**. - -## Force merging indexes - -To perform a force merge operation on two or more indexes, perform the following steps: - -1. Under **Index Management**, choose **Indices**. - -1. Select the indexes you want to force merge. - -1. Choose **Actions**, and then choose **Force merge**, as shown in the following image. - - ![Force Merge]({{site.url}}{{site.baseurl}}/images/admin-ui-index/forcemerge1.png) - -1. Under **Configure source index**, specify the indexes you want to force merge. - -1. Optionally, under **Advanced settings** you can to choose to **Flush indices** or **Only expunge delete** and then specify the **Max number of segments** to merge to as shown in the following image. - - ![Force Merge]({{site.url}}{{site.baseurl}}/images/admin-ui-index/forcemerge2.png) - -## Force merging data streams - -To perform a force merge operation on two or more indexes, perform the following steps: - -1. Under **Index Management**, choose **Data streams**. - -1. Select the data streams you want to force merge. - -1. Choose **Actions**, and then choose **Force merge**. - -1. Under **Configure source index**, specify the data streams you want to force merge. - -1. Optionally, under **Advanced settings** you can to choose to **Flush indices** or **Only expunge delete** and then specify the **Max number of segments** to merge to as shown in the following image. - - ![Force Merge]({{site.url}}{{site.baseurl}}/images/admin-ui-index/forcemerge2.png) diff --git a/_dashboards/im-dashboards/index.md b/_dashboards/im-dashboards/index.md index 8aaf5297d4d..fc031bf4f35 100644 --- a/_dashboards/im-dashboards/index.md +++ b/_dashboards/im-dashboards/index.md @@ -1,53 +1,49 @@ --- layout: default -title: Index management -nav_order: 100 -has_children: true +title: Index Management +nav_order: 30 redirect_from: - /dashboards/admin-ui-index/ - - /dashboards/im-dashboards/ + - /dashboards/admin-ui-index/index-management/ --- -# Index management +# Index Management in OpenSearch Dashboards + Introduced 2.5 {: .label .label-purple } -The Index Management interface in OpenSearch Dashboards provides a unified solution for managing common indexing and data stream operations. The interface allows you to perform create, read, update, and delete (CRUD) and mapping operations for indexes, index templates, and aliases instead of using REST APIs or YAML configurations for basic administrative operations and interventions, along with other operations such as open, close, reindex, shrink, and split indexes. The interface also provides you with the capabilities to run index status and data validation before submitting requests and compare changes with previously saved settings before making updates. - -An example of the interface is shown in the following image. - -![Index Management user interface]({{site.url}}{{site.baseurl}}/images/dashboards/index-management-UI.png) - -## Get started with index management using Dashboards - -**Step 1: Open Index Management** - -- Once you're in OpenSearch Dashboards, select **Index Management** from the **OpenSearch Plugins** main menu. Then select **Indices**. - -**Step 2: View indexes** - -- In the **Indices** interface you will see a list of existing indexes in your OpenSearch cluster. The list provides information such as index name, health state, document count, index size, and other relevant details. - -**Step 3: Create an index** - -- To create a new index, select the **Create index** button in the upper-right corner. You will be prompted to enter the index name and configure the index settings, such as number of shards and replicas. Fill in the required information and select **Create** to create the index. +Index Management in OpenSearch Dashboards provides an interface for creating, using, and managing indexes. In the OpenSearch **Index Management** web-based interface, you can perform most operations available in the [Index API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index/), including the following: -**Step 4: Delete an index** +- Create indexes and define their data mappings. +- Assign index sharding and replication. +- Perform management tasks, such as opening, closing, merging, splitting, and deleting indexes. +- Define state management policies to automate tasks such as log rollup and rollover, and assign those policies to indexes. +- Create templates, aliases, and notifications to use with index management policies. -- To delete an index, locate the index and select the checkbox next to it. Then select the **Actions** button and choose **Delete** from the dropdown list. Use caution when deleting indexes because this action is irreversible. +## Navigating the Index Management UI -**Step 5: Modify an index** +The following image shows how to navigate to the **Index Management** features. -- To modify the settings of an existing index, locate the index in the list and select its name. This takes you to the index details page. Here you can update settings such as the numbers of shards, replicas, and other advanced configurations. After making the desired changes, select **Save**. +Index Management panel -**Step 6: Refresh indexes** +- Select the **Index Management** _link_ (A) in the _navigation panel_ to view the Index Management features. +- The **Index Management** _panel_ (B) contains the index management features. +- The _breadcrumb_ display (C) shows which application or feature is active on the application page. -- To refresh an index, locate the index and select the checkbox next to it. Then select the **Actions** button and choose **Refresh** from the dropdown list. +Following is a short summary of the index management operations available in the **Index Management** panel. -**Step 7: Filter and search indexes** +- [**State management policies**]({{site.url}}{{site.baseurl}}/im-plugin/ism/dash-ism/): Create and modify policies that can automatically manage indexes. +- [**Policy managed indexes**]({{site.url}}{{site.baseurl}}/im-plugin/ism/dash-ism): View, remove, and change state management policies applied to indexes. +- [**Indexes**]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops): View all indexes. Create and manage indexes and perform operations on them. +- [**Data streams**]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/): View, create, and manage data streams. +- [**Templates**]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates): View, create, and manage templates to automate index and data stream creation. +- [**Aliases**]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/): View, create, and manage aliases. Aliases are virtual indexes that can incorporate multiple real indexes. +- [**Rollup jobs**]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/): Create and manage rollup jobs. Rollup jobs are like specialized transform jobs for compressing and archiving continuous indexes such as log data. +- [**Transform jobs**]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms): Create and manage transform jobs. Transform jobs append transformed data from one index to a second index on a schedule you define. +- [**Notification settings**]({{site.url}}{{site.baseurl}}/im-plugin/notifications/dash-notifications): Configure notifications via email or other channels for failed or completed index-related jobs. -- If you have a large number of indexes and want to filter or search for specific indexes, you can use the search bar located above the list of indexes. Enter the relevant keywords or filters to narrow the list of indexes. +## Next steps -**Step 8: Additional operations** +For more information about index management, including how to perform index maangement operations using the Index management API or the Index Management page in OpenSearch Dashboards, see [Managing indexes]({{site.url}}{{site.baseurl}}/im-plugin/). -- Index Management provides additional functionalities such as creating index patterns, managing lifecycle policies, and configuring index templates. These options are available in their respective sections of the Index Management interface. +Index management does not include data operations such as adding, updating, and querying documents. For an introduction to adding data to indexes, see [Ingest your data into OpenSearch]({{site.url}}{{site.baseurl}}/getting-started/ingest-data/). For information about using OpenSearch Dashboards to query your data, see [OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/). diff --git a/_dashboards/im-dashboards/rollover.md b/_dashboards/im-dashboards/rollover.md deleted file mode 100644 index d1510f14138..00000000000 --- a/_dashboards/im-dashboards/rollover.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -layout: default -title: Rollover -parent: Index management -nav_order: 40 -redirect_from: - - /dashboards/admin-ui-index/rollover/ ---- - -# Rollover -Introduced 2.6 -{: .label .label-purple } - -OpenSearch Dashboards allows you to perform an [index rollover]({{site.url}}{{site.baseurl}}/im-plugin/ism/error-prevention/index/#roll-over) operation with **Index Management**. - -## Data streams - -To perform a rollover operation on a data stream, perform the following steps: - -1. Under **Index Management**, choose **Data streams**. - -1. Choose **Actions**, and then choose **Roll over**, as shown in the following image. - - ![Roll over]({{site.url}}{{site.baseurl}}/images/admin-ui-index/rollover1.png) - -1. Under **Configure source**, select the source data stream on which you want to perform the rollover operation. - -1. Choose **Roll over**, as shown in the following image. - - ![Roll over]({{site.url}}{{site.baseurl}}/images/admin-ui-index/rollover3.png) - -## Aliases - -To perform a rollover operation on an alias, perform the following steps: - -1. Under **Index Management**, choose **Aliases**. - -1. Choose **Actions**, and then choose **Roll over**, as shown in the following image. - - ![Roll over]({{site.url}}{{site.baseurl}}/images/admin-ui-index/rollover2.png) - -1. Under **Configure source**, select the source alias on which you want to perform the rollover operation. - -1. If the alias does not contain a write index, you are prompted to assign a write index, as shown in the following image. - - ![Roll over]({{site.url}}{{site.baseurl}}/images/admin-ui-index/rollover4.png) - -1. Under **Configure a new rollover index** and on the **Define index** pane, specify an index name and an optional index alias. - -1. Under **Index settings**, specify the number of primary shards, the number of replicas, and the refresh interval, as shown in the following image. - - ![Roll over]({{site.url}}{{site.baseurl}}/images/admin-ui-index/rollover5.png) - -1. Choose **Roll over**. diff --git a/_dashboards/management/data-sources.md b/_dashboards/management/data-sources.md index 2873cff94c8..2ce0518436a 100644 --- a/_dashboards/management/data-sources.md +++ b/_dashboards/management/data-sources.md @@ -43,7 +43,7 @@ Alternatively, if you are running OpenSearch Dashboards 2.16 or later, go to **M ## Next steps - Learn about [managing index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/) through OpenSearch Dashboards. -- Learn about [indexing data using Index Management]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) through OpenSearch Dashboards. +- Learn about [Index Management]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) in OpenSearch Dashboards. - Learn about how to connect [multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/). - Learn about how to connect [OpenSearch and Amazon S3]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/) and [OpenSearch and Prometheus]({{site.url}}{{site.baseurl}}/dashboards/management/connect-prometheus/) using the OpenSearch Dashboards interface. - Learn about the [Integrations]({{site.url}}{{site.baseurl}}/integrations/index/) plugin, which gives you the flexibility to use various data ingestion methods and connect data to OpenSearch Dashboards. diff --git a/_dashboards/management/index-patterns.md b/_dashboards/management/index-patterns.md index 27c707e652e..fe27c323e34 100644 --- a/_dashboards/management/index-patterns.md +++ b/_dashboards/management/index-patterns.md @@ -7,9 +7,12 @@ nav_order: 10 # Index patterns -Index patterns are essential for accessing OpenSearch data. An _index pattern_ references one or more indexes, data streams, or index aliases. For example, an index pattern can point you to your log data from yesterday or all indexes that contain that data. +Index patterns are essential for accessing OpenSearch data. An _index pattern_ references one or more indexes, data streams, or index aliases. For example, an index pattern can point you to your log data from yesterday or all indexes that contain that data. -If you store data in multiple indexes, creating an index pattern enables your visualizations to retrieve data from all indexes that match the index pattern. You need to create index patterns to define how data is retrieved and fields are formatted so that you can query, search, and display data. +If you store data in multiple indexes, creating an index pattern enables your visualizations to retrieve data from all indexes that match the index pattern. You need to create index patterns to define how data is retrieved and fields are formatted so that you can query, search, and display data. + +Index patterns and aliases seem similar in that they both provide a way to reference multiple indexes. However, they serve different purposes. Index patterns reference indexes, as well as data streams _and_ aliases, in order to perform searches and other data operations. Aliases exist to enable index management. See [Aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). +{: .important} diff --git a/_getting-started/communicate.md b/_getting-started/communicate.md index 8166bc28e27..48b572c83e2 100644 --- a/_getting-started/communicate.md +++ b/_getting-started/communicate.md @@ -7,7 +7,11 @@ description: "Learn how to communicate with OpenSearch using the REST API to ind # Communicate with OpenSearch -You can communicate with OpenSearch using the REST API or one of the OpenSearch language clients. This page introduces the OpenSearch REST API. If you need to communicate with OpenSearch in your programming language, see the [Clients]({{site.url}}{{site.baseurl}}/clients/) section for a list of available clients. +You can communicate with OpenSearch using the REST API, one of the OpenSearch language clients, or the OpenSearch Dashboards web interface. This page introduces the OpenSearch REST API. + +To communicate with OpenSearch in your programming language, see the [Clients]({{site.url}}{{site.baseurl}}/clients/) section for a list of available clients. + +To use the OpenSearch Dashboards web interface, see [OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/). ## OpenSearch REST API @@ -101,7 +105,9 @@ PUT /students/_doc/1 Once you send the preceding request, OpenSearch creates an index called `students` and stores the ingested document in the index. If you don't provide an ID for your document, OpenSearch generates a document ID. In the preceding request, the document ID is specified as the student ID (`1`). -To learn more about indexing, see [Managing indexes]({{site.url}}{{site.baseurl}}/im-plugin/). +To learn more about how to index documents, see [Ingest your data into OpenSearch]({{site.url}}{{site.baseurl}}/getting-started/ingest-data/). + +To learn how to manage indexes, see [Managing indexes]({{site.url}}{{site.baseurl}}/im-plugin/). ## Dynamic mapping diff --git a/_getting-started/concepts.md b/_getting-started/concepts.md index 2b70faf9f61..95a3dfdcd6a 100644 --- a/_getting-started/concepts.md +++ b/_getting-started/concepts.md @@ -41,6 +41,14 @@ This page defines key terms and concepts related to OpenSearch. - [***Bulk indexing***]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/): The process of indexing multiple documents in a single request. - [***Upsert***]({{site.url}}{{site.baseurl}}/api-reference/document-apis/update-document/#upsert): An operation that updates a document if it already exists or inserts a new document if it does not. +## Index management + +- [***Index management***]({{site.url}}{{site.baseurl}}/im-plugin/): OpenSearch Dashboard features and operations that support indexes. Most of these features were initially developed as _ISM plugin_ features. Features include index state, policy, alias, and transform functionality. They also include creation, update, and deletion of indexes and operational assignments such as sharding and replication. +- [***Index operation***]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/index/): An action affecting an index that can be initiated using an API or the OpenSearch Dashboards interface. +- [***Index state management (ISM)***]({{site.url}}{{site.baseurl}}/im-plugin/ism/): Index management features that support policies and policy-managed indexes. +- [***State management policy***]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/), ***index policy***, or ***policy***: An object consisting of states, actions, and transitions, used to automatically manage indexes, for example in log rotation and compression. + + ## Text analysis - [***Text analysis***]({{site.url}}{{site.baseurl}}/analyzers/): A process of splitting the unstructured free text content of a document into a sequence of terms, which are then stored in an inverted index. diff --git a/_getting-started/index-data.md b/_getting-started/index-data.md new file mode 100644 index 00000000000..0e71d760991 --- /dev/null +++ b/_getting-started/index-data.md @@ -0,0 +1,278 @@ +--- +layout: default +title: Index data +nav_order: 45 +--- + + +# Indexing your data + +Before you can search data, you must *index* it. Indexing is the method by which search engines organize data for fast retrieval. The resulting structure is called, fittingly, an index. + +In OpenSearch, the basic unit of data is a JSON *document*. Within an index, OpenSearch identifies each document using a unique ID. + +You perform data operations, including indexing a document, using the document APIs. See [Document API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/index/). + + +## Naming restrictions for indexes + +OpenSearch indexes have the following naming restrictions: + +- All letters must be lowercase. +- Index names can't begin with underscores (`_`) or hyphens (`-`). +- Index names can't contain spaces, commas, or the following characters: + + `:`, `"`, `*`, `+`, `/`, `\`, `|`, `?`, `#`, `>`, or `<` + + +## Adding data to an index + +A request sent to the Index API appears as follows: + +```json +PUT {index}/_doc/{id} +{ "A JSON": "document" } +``` + +A request to the `_bulk` API looks a little different, because you specify the index and ID in the bulk data: + +```json +POST _bulk +{ "index": { "_index": "", "_id": "" } } +{ "A JSON": "document" } +``` + +Bulk data must conform to a specific format, which requires a newline character (`\n`) at the end of every line, including the last line. This is the basic format: + +``` +Action and metadata\n +Optional document\n +Action and metadata\n +Optional document\n +``` + +The document is optional, because `delete` actions don't require a document. The other actions (`index`, `create`, and `update`) all require a document. If you specifically want the action to fail if the document already exists, use the `create` action instead of the `index` action. +{: .note } + +To index bulk data using the `curl` command, navigate to the folder where you have your file saved and run the following command: + +```json +curl -H "Content-Type: application/x-ndjson" -POST https://localhost:9200/data/_bulk -u 'admin:admin' --insecure --data-binary "@data.json" +``` + +If any one of the actions in the `_bulk` API fail, OpenSearch continues to execute the other actions. Examine the `items` array in the response to figure out what went wrong. The entries in the `items` array are in the same order as the actions specified in the request. + +OpenSearch automatically creates an index when you add a document to an index that doesn't already exist. It also automatically generates an ID if you don't specify an ID in the request. This simple example automatically creates the movies index, indexes the document, and assigns it a unique ID: + +```json +POST movies/_doc +{ "title": "Spirited Away" } +``` + +Automatic ID generation has a clear downside: because the indexing request didn't specify a document ID, you can't easily update the document at a later time. Also, if you run this request 10 times, OpenSearch indexes this document as 10 different documents with unique IDs. To specify an ID of 1, use the following request (note the use of PUT instead of POST): + +```json +PUT movies/_doc/1 +{ "title": "Spirited Away" } +``` + +Because you must specify an ID, if you run this command 10 times, you still have just one document indexed with the `_version` field incremented to 10. + +Indexes default to one primary shard and one replica. If you want to specify non-default settings, create the index before adding documents: + +```json +PUT more-movies +{ "settings": { "number_of_shards": 6, "number_of_replicas": 2 } } +``` + + +## Reading data + +After you index a document, you can retrieve it by sending a GET request to the same endpoint that you used for indexing: + +```json +GET movies/_doc/1 + +{ + "_index" : "movies", + "_type" : "_doc", + "_id" : "1", + "_version" : 1, + "_seq_no" : 0, + "_primary_term" : 1, + "found" : true, + "_source" : { + "title" : "Spirited Away" + } +} +``` + +You can see the document in the `_source` object. If the document is not found, the `found` key is `false` and the `_source` object is not part of the response. + +To retrieve multiple documents with a single command, use the `_mget` operation. +The format for retrieving multiple documents is similar to the `_bulk` operation, where you must specify the index and ID in the request body: + +```json +GET _mget +{ + "docs": [ + { + "_index": "", + "_id": "" + }, + { + "_index": "", + "_id": "" + } + ] +} +``` + +To only return specific fields in a document: + +```json +GET _mget +{ + "docs": [ + { + "_index": "", + "_id": "", + "_source": "field1" + }, + { + "_index": "", + "_id": "", + "_source": "field2" + } + ] +} +``` + +To check if a document exists: + +```json +HEAD movies/_doc/{doc-id} +``` + +If the document exists, you get back a `200 OK` response, and if it doesn't, you get back a `404 - Not Found` error. + +## Updating data + +To update existing fields or to add new fields, send a POST request to the `_update` operation with your changes in a `doc` object: + +```json +POST movies/_update/1 +{ + "doc": { + "title": "Castle in the Sky", + "genre": ["Animation", "Fantasy"] + } +} +``` + +Note the updated `title` field and new `genre` field: + +```json +GET movies/_doc/1 + +{ + "_index" : "movies", + "_type" : "_doc", + "_id" : "1", + "_version" : 2, + "_seq_no" : 1, + "_primary_term" : 1, + "found" : true, + "_source" : { + "title" : "Castle in the Sky", + "genre" : [ + "Animation", + "Fantasy" + ] + } +} +``` + +The document also has an incremented `_version` field. Use this field to keep track of how many times a document is updated. + +POST requests make partial updates to documents. To altogether replace a document, use a PUT request: + +```json +PUT movies/_doc/1 +{ + "title": "Spirited Away" +} +``` + +The document with ID of 1 will contain only the `title` field, because the entire document will be replaced with the document indexed in this PUT request. + +Use the `upsert` object to conditionally update documents based on whether they already exist. Here, if the document exists, its `title` field changes to `Castle in the Sky`. If it doesn't, OpenSearch indexes the document in the `upsert` object. + +```json +POST movies/_update/2 +{ + "doc": { + "title": "Castle in the Sky" + }, + "upsert": { + "title": "Only Yesterday", + "genre": ["Animation", "Fantasy"], + "date": 1993 + } +} +``` + +### Example response + +```json +{ + "_index" : "movies", + "_type" : "_doc", + "_id" : "2", + "_version" : 2, + "result" : "updated", + "_shards" : { + "total" : 2, + "successful" : 1, + "failed" : 0 + }, + "_seq_no" : 3, + "_primary_term" : 1 +} +``` + +Each update operation for a document has a unique combination of the `_seq_no` and `_primary_term` values. + +OpenSearch first writes your updates to the primary shard and then sends this change to all the replica shards. An uncommon issue can occur if multiple users of your OpenSearch-based application make updates to existing documents in the same index. In this situation, another user can read and update a document from a replica before it receives your update from the primary shard. Your update operation then ends up updating an older version of the document. In the best case, you and the other user make the same changes, and the document remains accurate. In the worst case, the document now contains out-of-date information. + +To prevent this situation, use the `_seq_no` and `_primary_term` values in the request header: + +```json +POST movies/_update/2?if_seq_no=3&if_primary_term=1 +{ + "doc": { + "title": "Castle in the Sky", + "genre": ["Animation", "Fantasy"] + } +} +``` + +If the document is updated after we retrieved it, the `_seq_no` and `_primary_term` values are different and our update operation fails with a `409 — Conflict` error. + +When using the `_bulk` API, specify the `_seq_no` and `_primary_term` values within the action metadata. + +## Deleting data + +To delete a document from an index, use a DELETE request: + +```json +DELETE movies/_doc/1 +``` + +The DELETE operation increments the `_version` field. If you add the document back to the same ID, the `_version` field increments again. This behavior occurs because OpenSearch deletes the document `_source`, but retains its metadata. + +## Next steps + +- Use Index Management (IM) to automate recurring index management activities and reduce storage costs. For more information, see [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index). + +- For instructions on how to reindex data, see [Reindex data]({{site.url}}{{site.baseurl}}/im-plugin/reindex/index/). diff --git a/_getting-started/ingest-data.md b/_getting-started/ingest-data.md index 060f8868dfe..ed0aaccd915 100644 --- a/_getting-started/ingest-data.md +++ b/_getting-started/ingest-data.md @@ -14,6 +14,12 @@ There are several ways to ingest data into OpenSearch: - Use Data Prepper---an OpenSearch server-side data collector that can enrich data for downstream analysis and visualization. For more information, see [Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/). - Use other ingestion tools. For more information, see [OpenSearch tools]({{site.url}}{{site.baseurl}}/tools/). +There is currently no way to index data using OpenSearch Dashboards. + +For situations in which new data arrives incrementally (for example, customer orders from a small business), you might use the Index API to add documents individually as they arrive. For situations in which the data flow is less frequent (for example, weekly updates to a marketing website), you might prefer to generate a file and send it to the `_bulk` API. For large numbers of documents, lumping requests together and using the `_bulk` API offers superior performance. If your documents are exceptionally large, however, you might need to index them individually. + +When indexing documents, the document `_id` must be 512 bytes or less in size. + ## Bulk indexing To index documents in bulk, you can use the [Bulk API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/). For example, if you want to index several documents into the `students` index, send the following request: diff --git a/_im-plugin/index-alias.md b/_im-plugin/aliases/api-aliases.md similarity index 53% rename from _im-plugin/index-alias.md rename to _im-plugin/aliases/api-aliases.md index 8840d5666b1..f6f0d8dbb76 100644 --- a/_im-plugin/index-alias.md +++ b/_im-plugin/aliases/api-aliases.md @@ -1,32 +1,24 @@ --- layout: default -title: Index aliases -nav_order: 5 -redirect_from: - - /opensearch/index-alias/ +title: Aliases API +parent: Index aliases +nav_order: 20 --- -# Index aliases -If your data is spread across multiple indexes, rather than keeping track of which indexes to query, you can create an _alias_ and query it instead. An alias is a virtual index name that can point to one or more indexes. Aliases provide a flexible way to manage your data without changing your application code. +# Using the Aliases API -Index aliases are useful in several scenarios. You can use them to maintain a consistent query endpoint while rotating daily or monthly log indexes, switch between different data sets for A/B testing, and manage environments with aliases such as `production-data` and `staging-data`. For example, if you're storing logs into indexes based on the month and you frequently query the logs for the previous two months, you can create a `last_2_months` alias and update the indexes it points to each month. Aliases also help during data migrations, allowing you to transition gradually from old to new index structures without interrupting queries. +This page presents the following examples of how to use the Aliases API to create and manage aliases: -Because you can change the indexes an alias points to at any time, referring to indexes using aliases in your applications allows you to reindex your data without any downtime. +- [Creating a simple alias](#creating-a-simple-alias) +- [Switching an alias to a different index](#switching-an-alias-to-a-different-index) +- [Pointing an alias to multiple indexes](#pointing-an-alias-to-multiple-indexes) +- [Creating an alias during index creation](#creating-a-simple-alias) +- [Filtered aliases](#filtered-aliases) +- [Write indexes for multi-index aliases](#write-indexes-for-multi-index-aliases) -Aliases provide several key benefits: +For a complete description of every Alias API, see [Alias APIs]({{site.url}}{{site.baseurl}}/api-reference/alias/index/). -- Switch between indexes without interrupting client applications, enabling zero-downtime operations. -- Group related indexes under a single logical name for flexible data organization. -- Use routing and filtering to optimize query performance. -- Applications can reference stable alias names instead of changing index names, simplifying application logic. - -When working with aliases, keep in mind these important behaviors: - -- All alias changes happen atomically—there's never a moment when an alias points to an unintended set of indexes. -- When using wildcard patterns, aliases capture indexes that match at creation time and don't automatically include new indexes created later. -- Writing to an alias that points to multiple indexes requires designating a write index. -- Filtered aliases automatically apply their filters to all search, count, and delete by query operations. ## Creating a simple alias @@ -47,6 +39,7 @@ POST /_aliases ``` {% include copy-curl.html %} + ## Switching an alias to a different index You can atomically switch an alias from one index to another: @@ -72,6 +65,7 @@ POST /_aliases ``` {% include copy-curl.html %} + ## Pointing an alias to multiple indexes An alias can point to multiple indexes for broader queries: @@ -91,6 +85,7 @@ POST /_aliases ``` {% include copy-curl.html %} + ## Creating an alias during index creation You can add an alias when creating an index: @@ -106,6 +101,7 @@ PUT /logs-2024-03 ``` {% include copy-curl.html %} + ## Filtered aliases Create different "views" of the same data using filters: @@ -130,6 +126,7 @@ POST /_aliases ``` {% include copy-curl.html %} + ## Write indexes for multi-index aliases When an alias points to multiple indexes, designate one as the write index: @@ -156,6 +153,7 @@ POST /_aliases ``` {% include copy-curl.html %} + ## API reference The following table provides commonly used alias commands. diff --git a/_im-plugin/aliases/dash-aliases.md b/_im-plugin/aliases/dash-aliases.md new file mode 100644 index 00000000000..fa1aa976b3e --- /dev/null +++ b/_im-plugin/aliases/dash-aliases.md @@ -0,0 +1,222 @@ +--- +layout: default +title: Managing aliases with Dashboards +parent: Index aliases +nav_order: 10 +--- + + +# Managing aliases with OpenSearch Dashboards + +An alias is a virtual index name that can point to one or more indexes. If your data is spread across multiple indexes, rather than keeping track of which indexes to query, you can create an alias and query it instead. See [Index aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/). + +This page shows to use OpenSearch Dashboards to create and manage aliases. It includes the following procedures: + +- [Viewing aliases](#viewing-aliases) +- [Creating an alias](#creating-an-alias) +- [Editing an alias](#editing-an-alias) +- [Deleting an alias](#deleting-an-alias) +- [Refreshing an alias](#refreshing-aliases) +- [Flushing an alias](#flushing-aliases) +- [Clearing an alias cache](#clearing-an-alias-cache) +- [Rolling over an alias](#rolling-over-an-alias) +- [Force merging aliases](#force-merging-aliases) + + +## Viewing aliases + +You can view the list of aliases on your cluster. + +To view the list of indexes in your cluster, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. Select **Aliases** to view a list of existing aliases in your OpenSearch cluster. The list provides the following information about each index: + + - **Alias name**: The name of the index. + - **Write index**: The writable backing index for the alias, if any. + - **Index name**: A list of backing indexes for the alias. + + +## Creating an alias + +To create an alias, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. Select **Aliases** in the **Index Management** navigation panel. + +1. Choose **Create alias**. + +1. In the **Create alias** dialog, enter or select one or more indexes and index patterns in the **Indexes or index patterns** combo box. + +1. Choose **Create alias**. + + +## Editing an alias + +To edit an alias, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. Select **Aliases** in the **Index Management** navigation panel. + +1. In the *Aliases* table, select the alias you want to edit from the **Alias name** column. + +1. In the **Update alias** dialog, delete, enter, or select one or more indexes and index patterns in the **Indexes or index patterns** combo box. + + You cannot change the name of an existing alias. + {: .note} + +1. Choose **Save changes**. + +## Deleting an alias + +To delete an alias, follow these steps: + +Deleted aliases are not recoverable. +{: .warning} + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. Select **Aliases** in the **Index Management** navigation panel. + +1. In the *Aliases* table, select the checkboxes for one or more aliases that you want to delete. + +1. Select **Actions**. + +1. In the **Actions** drop-down, select **Delete**. + +1. In the Delete aliases confirmation dialog, type `delete` in the text box. + +1. Select **Delete**. + + +## Refreshing aliases + +Refreshing an alias makes new operations (since the last refresh) visible to search operations. See the description in [Refresh API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/refresh/). + +The refresh operation can be applied only to open indexes associated with the specified aliases. + +To refresh one or more aliases, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. Select **Aliases** in the **Index Management** navigation panel. + +1. (Optional) In the **Aliases** table, select the checkboxes for one or more aliases that you want to refresh. + + If you do not select any aliases, all aliases are refreshed. + +1. Select the **Actions** button at the upper right of the Aliases panel. + +1. Select **Refresh** from the drop-down. + +1. In the confirmation dialog, select the **Refresh** button. + + +## Flushing aliases + +The flush operation performs a Lucene commit, writing segments to disk and starting a new transaction log. See the description in [Flush API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/flush/). + +The flush operation can be applied only to open indexes associated with the specified aliases. + +To flush one or more aliases, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. Select **Aliases** in the **Index Management** navigation panel. + +1. (Optional) In the **Aliases** table, select the checkboxes for one or more aliases that you want to flush. + + If you do not select any aliases, all aliases are flushed. + +1. Select the **Actions** button at the upper right of the **Aliases** panel. + +1. Select **Flush** from the drop-down. + +1. In the confirmation dialog, select the **Flush** button. + + +## Clearing an alias cache + +The clear cache API operation clears the caches of one or more indexes. For data streams, the API clears the caches of the stream’s backing indexes. See the description in [Clear Cache API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/clear-index-cache/). + +The clear cache operation can be applied only to open indexes associated with the specified aliases. + +To clear the caches of the indexes in one or more aliases, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. Select **Aliases** in the **Index Management** navigation panel. + +1. (Optional) In the **Aliases** table, select the checkboxes for one or more aliases for which you want to clear the caches. + + If you do not select any aliases, all aliases are cleared. + +1. Select the **Actions** button at the upper right of the **Aliases** panel. + +1. Select **Clear cache** from the drop-down. + +1. In the confirmation dialog, select the **Clear cache** button. + + +## Rolling over an alias +Introduced 2.6 +{: .label .label-purple } + +To perform a rollover operation on an alias, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Alias**. + +1. Choose **Actions**. + +1. From the **Actions** drop-down, choose **Roll over**. + +1. In the **Configure source** panel, select the source alias on which you want to perform the rollover operation from the **Select an alias or data stream** drop-down. + + The panel displays the alias's write index under the heading **Assigned source index**. + +1. In the Configure new rollover index panel, follow these steps to configure the new writing index for the rollover: + + 1. (Optional) Import the configuration from the old write index. + + To use the same configuration as the old write index, select the **Import from old write index** button. + + 1. Enter a name for the new index in the **Index name** box. + + 1. Enter (or modify) the definition, settings, and mapping for the new write index. See the description of definitions, settings, and mappings in [Creating an index]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops/). + +1. Select the **Roll over** button at the lower right of the page. + + Note that the write index has changed the **Backing indexes** column in the **Aliases** table. + + +## Force merging aliases + +You can force-merge the backing indexes of one or more aliases by applying force-merge to the aliases. See [Force merge API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/force-merge/). + +To perform a force merge operation on one or more aliases, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Aliases**. + +1. Choose **Actions**. + +1. From the **Actions** drop-down, choose **Force merge**. + +1. In the **Configure source index** panel, enter or select one or more aliases containing the backing indexes you want to force merge in the **Select source indexes or data streams** combo box. + +1. (Optional) Select {::nomarkdown}expand icon{:/} (expand) icon in the **Advanced settings** panel. + + 1. Set one or more advanced options: + + - **Index segments**: Select **Manually set number of segments** to specify the number of segments to merge to. To fully merge indexes, set the number of segments to 1. + - Select **Flush indexes** to flush indexes after the force merge. See the description in [Flush API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/flush/). + - Select **Remove deleted documents** to expunge deleted documents. See [Force merge API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/force-merge/). + - **Notifications**: You can set notifications for the force merge depending on outcome. Select one or both of **Has failed / timed out** or **Has completed**. + +1. Choose the **Force merge** button in the lower right of the **Force merge** page. diff --git a/_im-plugin/aliases/index.md b/_im-plugin/aliases/index.md new file mode 100644 index 00000000000..aeb87efc143 --- /dev/null +++ b/_im-plugin/aliases/index.md @@ -0,0 +1,57 @@ +--- +layout: default +title: Index aliases +nav_order: 40 +has_children: true +has_toc: false +redirect_from: + - /opensearch/index-alias/ + - im-plugin/index-alias/ +--- + + +# Index aliases + +An alias is a virtual index name that can point to one or more indexes. If your data is spread across multiple indexes, you can create an alias and query it instead. + +Aliases provide the following capabilities: + +- Group related indexes under a single logical name for flexible data organization. +- Use routing and filtering to optimize query performance. +- Refer to stable alias names, simplifying application logic. +- Switch between indexes without interrupting client applications, enabling zero-downtime operations. + +These capabilities make aliases useful in several scenarios. For example, you can: + +- Switch between different data sets for A/B testing. +- Migrate data between index structures without interrupting queries. +- Manage environments with aliases such as `production-data` and `staging-data`. +- Maintain a consistent query endpoint while rotating daily or monthly log indexes. + +For example, if you store logs in indexes based on the month and you frequently query the logs for the previous two months, you can create a `last_2_months` alias and update the indexes it points to each month. + +When working with aliases, keep in mind these important behaviors: + +- All alias changes happen atomically. There's never a moment when an alias points to an unintended set of indexes. +- When using wildcard patterns, aliases capture indexes that match at creation time. They don't automatically include new indexes created later. +- Writing to an alias that points to multiple indexes requires designating a write index. +- Filtered aliases automatically apply their filters to all search, count, and delete by query operations. + +To manage aliases with the Alias API, see [Using the Aliases API]({{site.url}}{{site.baseurl}}/im-plugin/aliases/api-aliases/). + +To manage aliases using OpenSearch Dashboards, see [Managing aliases with OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/). + +The following table provides links to instructions using both OpenSearch Dashboards and the OpenSearch APIs. + +OpenSource Dashboards | Alias API +:-- | :-- +[Clearing an alias cache]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/#clearing-an-alias-cache) | [Clear Cache API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/clear-index-cache//) +[Creating an alias]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/#creating-an-alias) | [Create Or Update Index Alias API]({{site.url}}{{site.baseurl}}/api-reference/alias/create-alias/) +[Deleting an alias]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/#deleting-an-alias) | [Delete Index Alias API]({{site.url}}{{site.baseurl}}/api-reference/alias/delete-alias/) +[Editing an alias]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/#editing-an-alias) | [Create Or Update Index Alias API]({{site.url}}{{site.baseurl}}/api-reference/alias/create-alias/) +[Flushing aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/#flushing-aliases) | [Flush API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/flush/) +[Force merging aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/#force-merging-aliases) | [Force merging multiple indexes]({{site.url}}{{site.baseurl}}/api-reference/index-apis/force-merge/#force-merging-multiple-indexes) +[Refreshing aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/#refreshing-aliases) | [Refresh Index API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/refresh/) +[Rolling over an alias]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/#rolling-over-an-alias) | [Rollover types]({{site.url}}{{site.baseurl}}/api-reference/index-apis/rollover/#rollover-types) + + \ No newline at end of file diff --git a/_im-plugin/data-streams/api-datastream.md b/_im-plugin/data-streams/api-datastream.md new file mode 100644 index 00000000000..75b4eabc8dd --- /dev/null +++ b/_im-plugin/data-streams/api-datastream.md @@ -0,0 +1,283 @@ +--- +layout: default +title: Data streams API +parent: Data streams +nav_order: 20 +redirect_from: + - /opensearch/data-streams/ +--- + + +## Data streams API + +You can use the index and data stream APIs to create and manage data streams, including the following operations: + +- [Creating an index template](#creating-an-index-template) +- [Creating a data stream](#creating-a-data-stream) +- [Viewing data streams](#viewing-data-streams) +- [Ingesting data into the data stream](#ingesting-data-into-the-data-stream) +- [Searching a data stream](#searching-a-data-stream) +- [Rolling over a data stream](#rolling-over-a-data-stream) +- [Deleting a data stream](#deleting-a-data-stream) + +### Creating an index template + +To create an index template for a data stream, specify a data stream in the definition, as follows: + + +```json +PUT _index_template/logs-template +{ + "index_patterns": [ + "my-data-stream", + "logs-*" + ], + "data_stream": {}, + "priority": 100 +} +``` + +The `data_stream` object indicates that it’s a data stream and not a regular index template. + +The index pattern should match the name of the data stream. + +A data stream must havew a timestamp type field. By default this field is named `@timestamp`, but you can define a custom timestamp field as a property in the `data_stream` object. You can also add index mappings and other settings here, just as you would for a regular index template, as follows: + +```json +PUT _index_template/logs-template-nginx +{ + "index_patterns": "logs-nginx", + "data_stream": { + "timestamp_field": { + "name": "request_time" + } + }, + "priority": 200, + "template": { + "settings": { + "number_of_shards": 1, + "number_of_replicas": 0 + } + } +} +``` + +An index named `logs-nginx` matches both the previously defined `logs-template` and the `logs-template-nginx` template. When an index matches multiple templates, OpenSearch selects the matching index template with the higher priority value, in this case `logs-template-nginx`. + +### Creating a data stream + +Use the [Data Stream API] to explicitly create a data stream, as in the following example. + +```json +PUT _data_stream/logs-redis +PUT _data_stream/logs-nginx +``` + +The Data Stream API initializes the first backing index. + +You can also directly start ingesting data without creating a data stream. + +Because there is a matching index template with a data_stream object, OpenSearch automatically creates the data stream: + +```json +POST logs-staging/_doc +{ + "message": "login attempt failed", + "@timestamp": "2013-03-01T00:00:00" +} +``` + +### Viewing data streams + +You can view a [single data stream](#viewing-a-data-stream), [data stream statistics](#viewing-data-stream-statistics), or [all data streams](#viewing-all-data-streams). + +#### Viewing a data stream + +To see information about a specific data stream, do the following: + +```json +GET _data_stream/logs-nginx +``` + +##### Example response + +```json +{ + "data_streams" : [ + { + "name" : "logs-nginx", + "timestamp_field" : { + "name" : "request_time" + }, + "indices" : [ + { + "index_name" : ".ds-logs-nginx-000001", + "index_uuid" : "-VhmuhrQQ6ipYCmBhn6vLw" + } + ], + "generation" : 1, + "status" : "GREEN", + "template" : "logs-template-nginx" + } + ] +} +``` + +The API returns the following information: +- The name of the timestamp field +- A list of the backing indexes +- The template that's used to create the data stream. +- The health of the data stream. This is the lowest health status of all its backing indexes. + +#### Viewing data stream statistics + +To view cluster about the data stream, use the `_stats` endpoint, as follows: + +```json +GET _data_stream/logs-nginx/_stats +``` + +##### Example response + +```json +{ + "_shards" : { + "total" : 1, + "successful" : 1, + "failed" : 0 + }, + "data_stream_count" : 1, + "backing_indices" : 1, + "total_store_size_bytes" : 208, + "data_streams" : [ + { + "data_stream" : "logs-nginx", + "backing_indices" : 1, + "store_size_bytes" : 208, + "maximum_timestamp" : 0 + } + ] +} +``` + +#### Viewing all data streams + +To see information about all data streams, use the following request: + +```json +GET _data_stream +``` + +### Ingesting data into the data stream + +To ingest data into a data stream, use the [Index Document APIs]({{site.url}}{{site.baseurl}}/api-reference/document-apis/index-document/) as follows: + +```json +POST logs-redis/_doc +{ + "message": "login attempt", + "@timestamp": "2013-03-01T00:00:00" +} +``` + +Make sure every document that you index has a timestamp field. Ingesting a document that doesn't have a timestamp field generates error. +{: .warning} + +### Searching a data stream + +You can search a data stream just like you search a regular index or an index alias. The search operation applies to all of the backing indexes (all data present in the stream). + +To search a data stream do the following: + +```json +GET logs-redis/_search +{ + "query": { + "match": { + "message": "login" + } + } +} +``` + +#### Example response + +```json +{ + "took" : 514, + "timed_out" : false, + "_shards" : { + "total" : 5, + "successful" : 5, + "skipped" : 0, + "failed" : 0 + }, + "hits" : { + "total" : { + "value" : 1, + "relation" : "eq" + }, + "max_score" : 0.2876821, + "hits" : [ + { + "_index" : ".ds-logs-redis-000001", + "_type" : "_doc", + "_id" : "-rhVmXoBL6BAVWH3mMpC", + "_score" : 0.2876821, + "_source" : { + "message" : "login attempt", + "@timestamp" : "2013-03-01T00:00:00" + } + } + ] + } +} +``` + +### Rolling over a data stream + +To perform manual rollover operation on the data stream: + +```json +POST logs-redis/_rollover +``` + +#### Example response + +```json +{ + "acknowledged" : true, + "shards_acknowledged" : true, + "old_index" : ".ds-logs-redis-000001", + "new_index" : ".ds-logs-redis-000002", + "rolled_over" : true, + "dry_run" : false, + "conditions" : { } +} +``` + +If you now perform a `GET` operation on the `logs-redis` data stream, you see that the generation ID is incremented from 1 to 2. + +You can also set up an [Index State Management (ISM) policy]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/) to automate the rollover process for the data stream. +The ISM policy is applied to the backing indexes at the time of their creation. When you associate a policy to a data stream, it only affects the future backing indexes of that data stream. + +You also don’t need to provide the `rollover_alias` setting, because the ISM policy infers this information from the backing index. + + +### Deleting a data stream + +The delete operation first deletes the backing indexes of a data stream and then deletes the data stream itself. + +To delete a data stream and all of its hidden backing indexes: + +```json +DELETE _data_stream/{name_of_data_stream} +``` + +You can use wildcards to delete more than one data stream. + +We recommend deleting data from a data stream using an [ISM policy]({{site.url}}{{site.baseurl}}/im-plugin/ism/). + +You can use [asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/index/), [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/), and [PPL]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) to query your data stream directly. + +You can use the Security plugin to define granular permissions for the data stream name. diff --git a/_im-plugin/data-streams/dash-datastream.md b/_im-plugin/data-streams/dash-datastream.md new file mode 100644 index 00000000000..02330b5437a --- /dev/null +++ b/_im-plugin/data-streams/dash-datastream.md @@ -0,0 +1,261 @@ +--- +layout: default +title: Data streams in OpenSearch Dashboards +parent: Data streams +nav_order: 30 +redirect_from: + - /dashboards/admin-ui-index/datastream/ + - /opensearch/data-streams/ +--- + +# Working with data streams in OpenSearch Dashboards +Introduced 2.6 +{: .label .label-purple } + +For information about data streams, see [Data streams]({{site.url}}{{site.baseurl}}im-plugin/data-streams/index/). + +This page shows to use OpenSearch Dashboards to create and manage data streams from the Index Management page. It includes the following procedures: + +- [Viewing data streams](#viewing-data-streams) +- [Creating a data stream](#creating-a-data-stream) +- [Deleting a data stream](#deleting-a-data-stream) +- [Rolling over a data stream](#rolling-over-a-data-stream) +- [Refreshing a data stream](#refreshing-a-data-stream) +- [Flushing a data stream](#flushing-a-data-stream) +- [Clearing a data stream cache](#clearing-a-data-stream-cache) +- [Force merging indexes](#force-merging-data-streams). + + +## Viewing data streams + +On the Index Management page, you can view a table of data streams, or you can view data streams in the Indexes table. These options present slightly different information about the data streams. + +You can view details of data streams and backing data streams by selecting them from the table. + +The following sections describe all data stream viewing options. + + +### Viewing a table of data streams + +To view a table of data streams, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. Select **Data streams** from the **Index Management** panel. + + The table provides the following information about each data stream: + + - **Data stream name**: The name of the data stream. + + - **Status**: One of three replication status codes, as follows: + - Green: All primary and replica shards are assigned. + - Yellow: At least one replica shard is not assigned. + - Red: At least one primary shard is not assigned. + + Status codes are the same as index Health codes in the Indexes table. + {: .note} + + - **Template**: The index template used to create the data stream. + + Unlike other indexes, data streams must be created using a template. + {: .note} + + - **Backing indexes count**: The number of backing indexes that contain data for the data stream. + + - **Total size**: The sum of all storage used for the index over all primary and replica shards. + + +### Viewing data streams in the Indexes table + +View data streams in the **Indexes** table. See [Viewing the list of indexes]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops/#viewing-the-list-of-indexes) for a description of the **Index** table contents. + +To view data streams in the **Indexes** table, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. From the **Index Management** panel, select **Indexes**. + +1. In the **Indexes** table header, select **Show data stream indexes**. + + The table displays entries for all backing data streams. An extra column, **Data stream**, appears in the table. The column contains the name of the data stream that the backing stream entry belongs to. + + The table header also now displays a **Data streams** drop-down. + +1. (Optional) From the **Data streams** drop-down, select one or more data stream names to filter the table. + + The **Indexes** table contains only backing data streams that belong to one of the data streams selected in the drop-down. + + +### Viewing a single data stream + +To view details of a single data stream, follow these steps: + +1. View the **Data stream** table as described in [Viewing a table of data streams](#viewing-a-table-of-data-streams). + +1. In the **Data stream name** column, select the name of the data stream that you want to view. + + The page displays two panels of details: + + - **Data stream details** shows the following information: + - Name of the data stream. + - The data stream's status (health). + - The template used to create the data stream. + - The number of backing indexes. + - The data stream's time field name. + - **Backing indexes** shows a table of the data stream's backing indexes. It contains the following columns: + - Index: The name of the backing index. + - Health: The health status of the backing index. + - Managed by policy: Whether the backing index is managed by an index state management (ISM) policy. + - Status: Whether the backing index is open or closed. + - Writing index: Whether the backing index is a writing index. + - Total size: The number of bytes used by the backing index. + - Size of primaries: The number of bytes used by the backing index's primary shards. + - Total documents: The number of documents contained in the backing index. + - Deleted documents: The number of documents deleted from the backing index. + - Primaries: The number of primary shards used by the backing index. + - Replicas: The number of replicas used by the backing index. + +### Viewing a single backing index + +To view details of a single backing index, follow these steps: + +1. View the indexes table as described in [Viewing data streams in the index table](#viewing-data-streams-in-the-indexes-table). + +1. In the **Index** column, select the name of the backing index that you want to view. + +1. The backing index is displayed in the same format as a regular index. See [Viewing a single index]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops/#viewing-index-details). + + +## Creating a data stream + +To create a data stream, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Data streams**. + +1. Choose **Create data stream**. + +1. In the **Define data stream** panel, begin entering a name in the **Data stream name** combo box that matches an index pattern in a data stream index template. + + As you type, the **Data stream name** combo box displays a drop-down containing a list of matching index patterns and corresponding index template names. + +1. Select one of the index patterns from the drop-down or continue typing the index pattern. + +1. Complete the data stream name with a unique string that matches the index pattern. + + The **Matching template** field populates with the name of the index template containing the index pattern. + + The fields in the **Inherited settings from template** panel are read-only. They display the data stream definition from the index template. The **Index settings** and **Index mapping** are inherited from the template. + +1. Choose the **Create data stream** button in the lower right of the **Create data stream** page. + + +## Deleting a data stream + +To delete a data stream, follow these steps: + +Deleted data streams are not recoverable. +{: .warning} + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Data streams**. + +1. In the **Data stream names** column, select the data stream that you want to delete. + +1. Choose **Actions** in the upper right of the **Data streams** panel. + +1. From the drop-down, choose **Delete**. + +1. In the confirmation dialog, type `delete` in the text box. + +1. Choose **Delete**. + + +## Rolling over a data stream + +To perform a rollover operation on a data stream, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Data streams**. + +1. Choose **Actions**. + +1. From the **Actions** drop-down, choose **Roll over**. + +1. In the **Configure source** panel, select the source data stream on which you want to perform the rollover operation from the **Select an alias or data stream** drop-down. + +1. Select **Roll over**. + +1. (Optional) From the **Data streams** panel, select the data stream that you rolled over. + + Note that there is a new backing index in the **Backing indexes** table on the data stream's details page. + + +## Refreshing a data stream + +Refreshing a data stream makes new operations (since the last refresh) visible to search operations. See the description in [Refresh API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/refresh/). + +The refresh operation can be applied only to open data streams. + +To refresh one or more data streams, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Data streams**. + +1. Select one or more data streams from the **Data streams** table. + +1. Select the **Actions** button at the upper right of the **Data streams** panel. + +1. Select **Refresh** from the drop-down. + +1. In the confirmation dialog, select the **Refresh** button. + + +## Flushing a data stream + +The flush operation performs a Lucene commit, writing segments to disk and starting a new transaction log. See the description in [Flush API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/flush/). + +The flush operation can be applied only to open data streams. + +To flush one or more data streams, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Data streams**. + +1. Select one or more data streams from the **Data streams** list. + +1. Select the **Actions** button at the upper right of the **Data streams** panel. + +1. Select **Flush** from the drop-down. + +1. In the confirmation dialog, select the **Flush** button. + + +## Clearing a data stream cache + +The clear cache operation clears all field, query, and request caches for one or more data streams. See the description in [Clear Cache API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/clear-index-cache/). + +The clear cache operation can be applied only to open data streams. + +To clear the caches of one or more data streams, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Data streams**. + +1. Select one or more data streams from the **Data streams** list. + +1. Select the **Actions** button at the upper right of the **Data streams** panel. + +1. Select **Clear cache** from the drop-down. + +1. In the confirmation dialog, select the **Clear cache** button. + +## Force merging data streams + +Force merging a data stream is identical to force merging an index. See [Force merging indexes]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-manage/#force-merging-indexes). diff --git a/_im-plugin/data-streams/index.md b/_im-plugin/data-streams/index.md new file mode 100644 index 00000000000..eee80459613 --- /dev/null +++ b/_im-plugin/data-streams/index.md @@ -0,0 +1,107 @@ +--- +layout: default +title: Data streams +nav_order: 40 +has_children: true +has_toc: false +redirect_from: + - /opensearch/datastreams/ + - /im-plugin/data-streams/ +--- + +# Data streams + +If you're ingesting continuously generated time-series data such as logs, events, and metrics into OpenSearch, you're likely in a scenario where the number of documents grows rapidly and you don't need to update older documents. + +A typical workflow to manage time-series data involves multiple steps, such as creating a rollover index alias, defining a write index, and defining common mappings and settings for the backing indexes. Data streams simplify this process and enforce features that support time-series data, such as requiring append-only data and a timestamp field. + +Internally, a data stream is composed of multiple backing indexes. Search requests are routed to all the backing indexes, while indexing requests are routed to the latest write index. You can implement [ISM]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/) policies to automatically handle index rollovers and deletions. + +## Data stream workflow + +Following is a typical sequence of use of data streams. These steps can be implemented using APIs or OpenSearch Dashboards. + +### Step 1: Create an index template + +To create a data stream, you first need to create an index template that configures a set of indexes as a data stream. + +To create an index template for a data stream with the index template API, see [Creating an index template]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#creating-an-index-template). + +To create an index template for a data stream in OpenSearch Dashboards, see [Creating an index template]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/). + + +### Step 2: Create a data stream + +After you create an index template, you can create a data stream. + +To create a data stream with the data stream API, see [Creating a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#creating-a-data-stream). + +To create an index template for a data stream in OpenSearch Dashboards, see [Creating an index template]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#creating-a-data-stream). + + +### Step 3: Ingest data into the data stream + +You can ingest data into a data stream as you would any index. + +To ingest data into a data stream using the Index document API, see [Ingesting data into the data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#ingesting-data-into-the-data-stream). + +To ingest data into a data stream using OpenSearch Dashboards, see [Ingest your data into OpenSearch]({{site.url}}{{site.baseurl}}/getting-started/ingest-data/). + +Make sure every document that you index has a timestamp field. If you try to ingest a document that doesn't have a timestamp field, you get an error. +{: warning} + + +### Step 4: Searching a data stream + +You can search a data stream just like you search a regular index or an index alias. +The search operation applies to all of the backing indexes (all data present in the stream). + +To search a data stream using the Search API, see [Searching a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#searching-a-data-stream). + +To search a data stream using OpenSearch Dashboards, see [Exploring data]({{site.url}}{{site.baseurl}}/dashboards/discover/index/). + + +### Step 5: Rollover a data stream + +A rollover operation creates a new backing index that becomes the data stream’s new write index. + +To roll over a data stream using the Rollover API, see [Rolling over a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#rolling-over-a-data-stream). + +To roll over a data stream using OpenSearch Dashboards, see [Rolling over a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#rolling-over-a-data-stream). + + +### Step 7: Delete a data stream + +When you delete a data stream, the delete operation first deletes the backing indexes and then deletes the data stream itself. + +Deleted data streams are not recoverable. +{: .warning} + +To delete a data stream and all of its backing indexes using the Data stream API, see [Deleting a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#deleting-a-data-stream). + +To delete a data stream and all of its backing indexes using OpenSearch Dashboards, see [Deleting a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#deleting-a-data-stream). + + +## All data stream operations + +To manage data streams with the Alias API, see [Data streams API]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/). + +To manage data streams using OpenSearch Dashboards, see [Working with data streams in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/). + +To learn about querying external data sources directly using the SQL plugin, see [Data source APIs]({{site.url}}{{site.baseurl}}/sql-and-ppl/sql-and-ppl-api/data-source-apis/index/) (experimental). + +The following table provides links to instructions using both OpenSearch Dashboards and the OpenSearch APIs. + +OpenSearch Dashboards | API +:-- | :-- +[Clearing a data stream cache]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#clearing-a-data-stream-cache) | [Clear cache]({{site.url}}{{site.baseurl}}/api-reference/index-apis/clear-index-cache/) +[Creating a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#creating-a-data-stream) | [Creating a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#creating-a-data-stream) +[Deleting a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#deleting-a-data-stream) | [Deleting a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#deleting-a-data-stream) +[Flushing a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#flushing-a-data-stream) | [Flush]({{site.url}}{{site.baseurl}}/api-reference/index-apis/flush/) +[Force merging data streams]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#force-merging-data-streams) | [Force merge]({{site.url}}{{site.baseurl}}/api-reference/index-apis/force-merge/) +[Refreshing a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#refreshing-a-data-stream) | [Refresh index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/refresh/) +[Rolling over a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#rolling-over-a-data-stream) | [Roll over index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/rollover/) +[Viewing a single backing index]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#viewing-a-single-backing-index) | [Get index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/get-index/) +[Viewing a single data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#viewing-a-single-data-stream) | [Viewing a data stream]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#viewing-a-data-stream) +[Viewing data stream statistics]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#viewing-a-single-data-stream) | [Viewing data stream statistics]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#viewing-data-stream-statistics) +[Viewing data streams]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#viewing-data-streams) | [Viewing all data streams]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/api-datastream/#viewing-all-data-streams) diff --git a/_im-plugin/append-only-index.md b/_im-plugin/index-ops/append-only-index.md similarity index 98% rename from _im-plugin/append-only-index.md rename to _im-plugin/index-ops/append-only-index.md index 2fd62ab4e20..0bb86ada79d 100644 --- a/_im-plugin/append-only-index.md +++ b/_im-plugin/index-ops/append-only-index.md @@ -1,7 +1,8 @@ --- layout: default title: Append-only index -nav_order: 15 +parent: Index operations +nav_order: 30 --- # Append-only index diff --git a/_im-plugin/index-ops/dash-manage.md b/_im-plugin/index-ops/dash-manage.md new file mode 100644 index 00000000000..e6b878f615f --- /dev/null +++ b/_im-plugin/index-ops/dash-manage.md @@ -0,0 +1,265 @@ +--- +layout: default +title: Index management operations +parent: Index operations +nav_order: 20 +redirect_from: + - /dashboards/admin-ui-index/index-management/ +--- + +# Managing indexes in OpenSearch Dashboards + +Introduced 2.5 +{: .label .label-purple } + +OpenSearch automatically performs such transaction- and memory-related operations as flushing index caches, refreshing data from the transaction log, and managing index caches. You can also perform these operations manually on one or more indexes from the **Index Management** page as described in the following procedures. + +In **Index Management** page of OpenSearch Dashboards, you can perform most of the operations available in the [Index API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index/), including the following: + +- [Refreshing an index](#refreshing-an-index) +- [Flushing indexes](#flushing-indexes) +- [Clearing an index cache](#clearing-an-index-cache) +- [Force merging indexes](#force-merging-indexes) +- [Shrinking an index](#shrinking-an-index) +- [Splitting an index](#splitting-an-index) + +You cannot clone an index in OpenSearch Dasboards. Use the [Clone Index API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/clone/) to clone an index. +{: .note} + +[UI topics](#ui-topics) briefly discusses considerations specific to performing these operations in OpenSearch Dashboards, including the following: + +- [Security](#security-integration) +- [Error messaging](#error-messages) +- [Checking the status of long-running operations](#checking-the-status-of-long-running-operations). + + +## Refreshing an index + +Refreshing an index makes new operations (since the last refresh) visible to search operations. See the description in [Refresh API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/refresh/). + +The refresh operation can be applied only to open indexes. + +To refresh one or more indexes, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. (Optional) Select one or more indexes from the **Indexes** list. + + If you do not select any indexes, all indexes are refreshed. + +1. Select the **Actions** button at the upper right of the Indexes panel. + +1. Select **Refresh** from the drop-down. + +1. In the confirmation dialog, select the **Refresh** button. + + +## Flushing indexes + +The flush operation performs a Lucene commit, writing segments to disk and starting a new transaction log. See the description in [Flush API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/flush/). + +The flush operation can be applied only to open indexes. + +To flush one or more indexes, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. (Optional) Select one or more indexes from the **Indexes** list. + + If you do not select any indexes, all indexes are flushed. + +1. Select the **Actions** button at the upper right of the Indexes panel. + +1. Select **Flush** from the drop-down. + +1. In the confirmation dialog, select the **Flush** button. + + +## Clearing an index cache + +The clear cache operation clears all field, query, and request caches for one or more indexes. See the description in [Clear Cache API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/clear-index-cache/). + +The clear cache operation can be applied only to open indexes. + +To clear the caches of one or more indexes, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. (Optional) Select one or more indexes from the **Indexes** list. + + If you do not select any indexes, all index caches are cleared. + +1. Select the **Actions** button at the upper right of the Indexes panel. + +1. Select **Clear cache** from the drop-down. + +1. In the confirmation dialog, select the **Clear cache** button. + + +## Force merging indexes + +In OpenSearch, indexed data is stored on segments. Periodically, the system merges smaller segments into larger ones in the background. Additionally, you can force a such a merge operation on one or more indexes at any time. See [Force merge API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/force-merge/) + +To perform a force merge operation on two or more indexes, data streams, or alias backing indexes, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Indexes**, **Data streams**, or **Aliases**. + +1. Choose **Actions**. + +1. From the **Actions** drop-down, choose **Force merge**. + +1. In the **Configure source index** panel, enter or select the indexes, backing indexes (for aliases), or data streams that you want to force merge in the **Select source indexes or data streams** combo box. + +1. (Optional) Select {::nomarkdown}expand icon{:/} (expand) icon in the **Advanced settings** panel. + + 1. Set one or more advanced options: + + - **Index segments**: Select **Manually set number of segments** to specify the number of segments to merge to. To fully merge indexes, set the number of segments to 1. + - Select **Flush indexes** to flush indexes after the force merge. See the description in [Flush API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/flush/). + - Select **Remove deleted documents** to expunge deleted documents. See [Force merge API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/force-merge/). + - **Notifications**: You can set notifications for the force merge depending on outcome. Select one or both of **Has failed / timed out** or **Has completed**. + +1. Choose the **Force merge** button in the lower right of the **Force merge** page. + + +## Shrinking an Index + +You can reduce the number of primary shards in an existing index by migrating data to a new index with fewer shards. See [Shrink Index API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/shrink-index/). + +### Prerequisites + +Before you can shrink an index, it must meet the following conditions: + +- The index must have index _blocks_ set to _write_. This is another way of saying that the index is read-only. See [Blocks API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/blocks/). +- A copy of every shard in the index (both primary and replica) must reside on the same node. Use shard allocation filtering to move shards to the same node. See [Shard allocation filtering]({{site.url}}{{site.baseurl}}/api-reference/index-apis/shard-allocation/). +- The cluster health status must be green. +- The target index’s primary shard count must be a factor of the source index’s primary shard count. For example, an index with 8 primary shards can be shrunk into 4, 2, or 1 primary shard. An index with a prime number of shards (such as 7) can only be shrunk into 1 primary shard. +- The target index must not already exist. +- The source index must have more primary shards than the target index. +- The source index must not contain more than 2,147,483,519 documents across all shards that will be merged into a single target shard, because this is the maximum number of documents a single Lucene shard can hold. +- The node handling the shrink process must have sufficient free disk space to accommodate a second copy of the existing index. + +### Procedure + +To shrink an index, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Indexes**. + +1. From the **Indexes** table, select the index you want to shrink. + +1. Choose the **Actions** button. + +1. From the drop-down list, select **Shrink**. + +1. In the **Configure target index** panel, enter a name for the target in the **Target index** name box. + +1. In the **Number of primary shards** box, enter the new shard count. + +1. In the **Number of replicas** box, enter the number of replicas. + +1. (Optional) In the **Index alias - _optional_** combo box, enter or select one or more aliases for the target index. + +1. (Optional) Select {::nomarkdown}expand icon{:/} (expand) **Advanced** to specify additional notifications. See [**Sending additional notifications**]({{site.url}}{{site.baseurl}}/im-plugin/notifications/dash-notifications/#sending-additional-notifications). + + +## Splitting an index + +The split index operation moves an existing read-only index into a new index, splitting each primary shard into a number of primary shards in the new index. This is useful when an index has outgrown its original shard count and needs additional data volume or query load capacity. + +For more about how a split works, see [Split Index API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/split/). + +### Prerequisites + +Before you can split an index, it must meet the following conditions: + +- The index must have index _blocks_ set to _write_. This is another way of saying that the index is read-only. See [Blocks API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/blocks/). The interface prompts you to block write operations when splitting an index in OpenSearch Dashboards. +- The cluster health status must be green. +- The target index must not already exist. +- The source index must have fewer primary shards than the target index. +- The number of primary shards in the target index must be a multiple of the source index’s primary shard count. For example, an index with 2 primary shards can be split into 4, 6, 8, or any other multiple of 2. An index with a single primary shard can be split into any number of shards. +- The node handling the split process must have sufficient free disk space to accommodate a second copy of the existing index. + + +### Procedure + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Indexes**. + +1. From the **Indexes** table, select the index you want to split. + +1. Choose the **Actions** button. + +1. From the drop-down list, select **Split**. + +1. In the **Configure target index** panel, enter a name for the target in the **Target index name** box. + +1. In the **Number of primary shards** box, enter the new shard count. + +1. In the **Number of replicas** box, enter the number of replicas. + +1. (Optional) In the **Index alias - _optional_** combo box, enter or select one or more aliases for the target index. + +1. (Optional) Select {::nomarkdown}expand icon{:/} (expand) **Advanced** to specify additional notifications. See [Sending additional notifications]({{site.url}}{{site.baseurl}}/im-plugin/notifications/dash-notifications/#sending-additional-notifications). + + + +## Creating jobs + +You can create scheduled jobs to help you maintain indexes. These include: + +- _Rollup jobs_ to compress older data. See [Index rollups]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/index/). +- _Transform jobs_ to create materialized data for queries and visualizations. See [Index transforms]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms/index/). + +To create a rollup job using OpenSearch Dashboards, see [Creating a rollup job]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/dash-rollup/). + +To create a transform job using OpenSearch Dashboards, see [Creating a transform job]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms/dash-transforms/). + + +### Next steps + +You can specify the index rollup job as an ISM action. This allows you to roll up an index after a certain event such as a rollover, index age reaching a certain point, index becoming read-only, and so on. You can also have rollover and index rollup jobs running in sequence, where the rollover first moves the current index to a warm node and then the index rollup job creates a new index with the minimized data on the hot node. + +You can enable and disable rollup jobs by choosing the corresponding buttons on the **Rollup Jobs** page. + +Likewise, you can specify the index transform job as an ISM action. You might for example periodically summarize data into a view that is represented by a dashboard on your cluster. + +See [Index state management with OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/ism/dash-ism/) for information about using ISM actions. + + +## UI topics + +The following sections describe differences between using OpenSearch Dashboards and using the OpenSearch APIs to perform index operations. + + +### Security integration + +Permission control is managed with existing [permissions]({{site.url}}{{site.baseurl}}/security-plugin/access-control/permissions/) or action groups that are enforced at the API level. There is currently no permission control available in OpenSearch Dashboards. If you have permission to access the ISM plugin, then you can view new pages. You can make changes if you have permission to run the related APIs. + +### Error messages + +If an ISM operation fails immediately, the plugin notifies you with an error message. + +For long-running operations, the plugin notifies you at the time of failure. + +You can also check the index status on the **Indexes** page at any time. See [Checking the status of long-running operations](#checking-the-status-of-long-running-operations). + + +## Checking the status of long-running operations + +Some index operations take time to complete (usually more than 30 seconds, but up to minutes or even hours). + +To check the status of a long-running index operation in OpenSearch Dashboards, follow these steps: + +1. In the Index Management panel, select **Indexes**. + +1. In the **Indexes** table, find the index that you performed the operation on. + +1. Look at the **Status** column to see the state of the operation. + + You can check the status of the reindex, shrink, and split operations because they are one-time, non-recursive operations. + diff --git a/_im-plugin/index-ops/dash-ops.md b/_im-plugin/index-ops/dash-ops.md new file mode 100644 index 00000000000..26422e122f6 --- /dev/null +++ b/_im-plugin/index-ops/dash-ops.md @@ -0,0 +1,241 @@ +--- +layout: default +title: Core index operations +parent: Index operations +nav_order: 10 +--- + + +# Core index operations in OpenSearch Dashboards + +Introduced 2.5 +{: .label .label-purple } + +You can use in OpenSearch Dashboards to perform the following core index operations: +- [Viewing indexes](#viewing-indexes) +- [Creating an index](#creating-an-index) +- [Creating an append-only index](#creating-an-append-only-index) +- [Modifying an index](#modifying-an-index) +- [Deleting an index](#deleting-an-index) +- [Opening and closing indexes](#opening-an-index) + + +## Viewing indexes + +You can view the list of indexes on your cluster. You can also select an index from the list to view detailed information about a single index. + + +### Viewing the list of indexes + +To view the list of indexes in your cluster, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. Select **Indexes** to view a list of existing indexes in your OpenSearch cluster. The list provides the following information about each index: + + - **Index**: The name of the index + - **Health**: One of three replication status codes, as follows: + - Green: All primary and replica shards are assigned. + - Yellow: At least one replica shard is not assigned. + - Red: At least one primary shard is not assigned. + - **Managed by policy**: Whether the index is contained (directly, or indirectly through an alias) in a policy. + - **Status**: Open or closed. + - **Total size**: The sum of all storage used for the index over all primary and replica shards. + - **Size of primaries**: The storage used for the index over all primary shards. + - **Total documents**: The number of documents contained in the index. + - **Deleted documents**: How many documents have been deleted from the index. + - **Primaries**: How many primary shards the index uses. + - **Replicas**: How many replica shards the index uses. + + +### Viewing index details + +To view details about an index, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. Select **Indexes** to view the list of existing indexes. + +1. In the Index column, select the name of the index you want to view. + + There might be more than one page of indexes in the list. + {: .note} + + The index page shows an Overview panel with metrics about the index's status. + + You can modify the index on this page. See [Modifying an index](#modifying-an-index). + +1. To return to the index list, select Index in the breadcrumb trail in the upper left next to the {::nomarkdown}home icon{:/} (home) icon. + + +## Creating an index + +To create a new index, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. On the **Index Management** page, select the **Create index** button in the upper-right corner of the **Indexes** panel. + + The **Create index** dialog contains three panels: + - **Define index** + - **Index settings** + - **Index mapping** (optional) + +1. In the **Define index** panel, do the following: + 1. Enter a name for the index in the **Index name** box. + 1. (Optional) Select one or more aliases for the index from the **Index alias - _optional_** combo box. Or, enter the name of a new alias you want to create. + +1. In the **Index settings** panel, do the following: + 1. In the **Number of primary shards** box, enter the number of primary shards for the index. + 1. In the **Number of replicas** box, enter the number of replicas for the index. + 1. In the **Refresh interval** box, enter how often the index should refresh to make new data available for search. The default interval is `1s`. + 1. (Optional) Select **Advanced settings** to upload configuration parameters in a flat JSON object. See [Index-level index settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index-settings/#index-level-index-settings) and [Query parameters `flat_settings`]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui). + +1. In the **Index mapping – _optional_** panel, do the following to add one or more index mappings. + 1. Select the **Visual editor** or the **JSON editor** button. We recommend using the Visual editor unless you're going to paste in an existing JSON mapping. + 1. Select **Add new field** or **Add new object**. + 1. In the **Field name** box, enter a name for the field or object. + 1. In the **Field type** drop-down, select a data type for the field. + 1. (Optional) If the field type is `object`, select the {::nomarkdown}plus icon{:/} (plus) icon one or more times to add nested objects or fields. + 1. Repeat the preceding four steps to add as many fields as needed. + +1. When all three panels have been completed, select the **Create** button to create the index. + + The system also creates any new aliases you specified in the **Define index** panel. + {: .note} + + +## Creating an append-only index + +To create an index with updates disabled, do the following: + +1. Create an index on the **Index Management** page of OpenSearch Dashboards as described in [Creating an Index](#creating-an-index). +1. In the **Index settings** panel, add the following setting specifier to the **Advanced settings**: + +```json + "index.append_only.enabled": "true" +``` +{% include copy.html %} + +After an index is created as append-only, it cannot be changed to another index type. +{: .warning} + + +## Modifying an index + +To modify an index, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. Select **Indexes** in the **Index Management** navigation panel. + +1. In the **Index** list, select the index name in the **Index** column. + + There might be more than one page of indexes in the list. + {: .note} + +1. Select the tab containing one of the following aspects of the index that you want to edit: + - **Settings** brings up the **Index settings** panel. + - **Mapping** brings up the **Index mappings** panel. + - **Alias** brings up the **Index alias** panel. + + 1. In the **Index settings** panel, change one or both of the following: + - Number of replicas + - Refresh interval + + You cannot change the number of primary shards of an existing index. + {: .note} + + - (Optional) In the **Index settings** panel, select **Advanced settings** to upload configuration parameters in a flat JSON object. See [Index-level index settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index-settings/#index-level-index-settings) and [Query parameters `flat_settings`]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui). + + 1. In the **Index mappings** panel you can add one or more new fields or objects. + + You cannot change the name or type of a field that already exists in an index. + + 1. In the **Index alias** panel, you can do any or all of the following: + - Enter one or more new aliases to the index. + - Select one or more existing aliases from the **Index alias – _optional_** drop-down. + - Remove one or more aliases from the index by selecting the {::nomarkdown}cross icon{:/} (cross) icon next to the alias name. + +1. When you are finished modifying the index in all three panels, select **Save**. + + +## Deleting an index + +To delete one or more indexes, follow these steps: + +Deleted indexes are not recoverable. +{: .warning} + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. Select **Indexes** in the **Index Management** navigation panel. + +1. Select the check box next every index that you want to delete. + + There might be more than one page of indexes in the list. + {: .note} + +1. Select the **Actions** button at the upper right of the Indexes panel. + +1. Select **Delete** from the drop-down. + +1. In the confirmation dialog, enter `delete` in the text box. + +1. Select the **Delete** button. + + +## Closing and opening indexes + +To update a static index setting, you must close the index, update the setting, and then reopen the index. Closing an index prevents reading from the index or performing data operations on it. See [Index-level index settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index-settings/#index-level-index-settings). + +You can open and close indexes from the **Index Management** page of OpenSearch Dashboards. + +### Closing an index + +To close one or more indexes, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. Select **Indexes** in the **Index Management** navigation panel. + +1. Select the check box next every index that you want to close. + + There might be more than one page of indexes in the list. + {: .note} + +1. Select the **Actions** button at the upper right of the Indexes panel. + +1. Select **Close** from the drop-down. + +1. In the confirmation dialog, enter `close` in the text box. + +1. Select the **Close** button. + +### Opening an index + +To open one or more closed indexes, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui) in OpenSearch Dashboards, select **Management** > **Index Management**. + +1. Select **Indexes** in the **Index Management** navigation panel. + +1. Select the check box next every index that you want to open. + + There might be more than one page of indexes in the list. + {: .note} + +1. Select the **Actions** button at the upper right of the Indexes panel. + +1. Select **Open** from the drop-down. + +1. Select the **Open** button. + + +## Next steps + +- To peform other index operations using OpenSearch Dashboards, see [Managing indexes in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-manage/). + +- To perform core index operations programmatically instead, use the [Core index APIs]({{site.url}}{{site.baseurl}}/api-reference/index-apis/core-index-apis/). + +Index state management (ISM) enables you to automate index management in your OpenSearch cluster. See [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/). diff --git a/_im-plugin/index-ops/index.md b/_im-plugin/index-ops/index.md new file mode 100644 index 00000000000..2b6a3eb2604 --- /dev/null +++ b/_im-plugin/index-ops/index.md @@ -0,0 +1,78 @@ +--- +layout: default +title: Index operations +nav_order: 20 +has_children: true +has_toc: false +--- + + +# Index operations + + +Index operations are used to manage indexes on a cluster. These include things like cloning, shrinking, scaling, flushing, and merging. The most basic of these are core index operations, used to create, delete, open, close, and get information about indexes. + +OpenSearch supports a number of operations on indexes. These operations are presented here in two categories, parallelling their organization in the OpenSearch [Index APIs]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index/): + +- [**Core index operations**]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/index/#core-index-operations) are the most basic operations, used to create, delete, open, close, and get information about indexes. +- [**Index management operations**]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/index/#index-management-operations) are used to manage indexes on a cluster. These include things like cloning, shrinking, scaling, flushing, and merging. + +In addition, the following index operations and features, documented in their own sections, support [index state management (ISM)]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/). + +- [Aliases]({{site.url}}{{site.baseurl}}/im-plugin/aliases/index/) +- [Data streams]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/index/) +- [Index rollups]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/index/) +- [Index transforms]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms/index/) +- [Notifications]({{site.url}}{{site.baseurl}}/im-plugin/notifications/index/) +- [Reindexing]({{site.url}}{{site.baseurl}}/im-plugin/reindex/index/) +- [Templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/index/) + + +## Core index operations + +Core index operations enable you to directly manage indexes in your OpenSearch cluster. These operations are also the building blocks for state managed indexes. + +OpenSearch provides following core index operations: + +- Retrieving information about an index or a group of indexes, including whether an index exists. +- Creating and deleting an index. +- Opening and closing an index. + +All of the core operations can be performed with the core index APIs or in OpenSearch Dashboards. + + +### In OpenSearch Dashboards + +For core index operations in OpenSearch Dashboards, see [Core dashboard operations]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops/). + + +### Using the APIs + +For core index operations using the Index APIs, see [Core index APIs]({{site.url}}{{site.baseurl}}/api-reference/index-apis/). + + +## Index management operations + +OpenSearch features many operations on indexes beyond the core index operations. Most of these operations can also be performed in OpenSearch Dashboards. + +Index operations include: + +- Operational tasks, such as clearing index caches and flushing, force-merging, and refreshing indexes. +- Segement and shard operations, such as splitting, shrinking, and scaling indexes. +- Querying information about indexes, including storage locations, statistics, and the status of long-running operations. +- Index manipulations like cloning, rolling over, and transforming indexes. + + +### In OpenSearch Dashboards + +For index operations in OpenSearch Dashboards, see [Managing indexes in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops/). + + +### Using the APIs + +For index operations using the Index APIs, see [Index APIs]({{site.url}}{{site.baseurl}}/api-reference/index-apis/). + + +## Next steps + +Index state management (ISM) enables you to automate index management in your OpenSearch cluster. See [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/). \ No newline at end of file diff --git a/_im-plugin/index-codecs.md b/_im-plugin/index-other/index-codecs.md similarity index 97% rename from _im-plugin/index-codecs.md rename to _im-plugin/index-other/index-codecs.md index 978bd5aa6da..be5352cbf29 100644 --- a/_im-plugin/index-codecs.md +++ b/_im-plugin/index-other/index-codecs.md @@ -1,8 +1,9 @@ --- layout: default title: Index codecs -nav_order: 3 -parent: Index settings +parent: Other components +grand_parent: Managing indexes +nav_order: 40 --- # Index codecs @@ -86,7 +87,7 @@ You can modify the value of `index.codec.qatmode` during the restore process by ### Reindexing -When you are performing a [reindex]({{site.url}}{{site.baseurl}}/im-plugin/reindex-data/) operation from a source index, the new segments created in the target index will have the properties of the codec settings of the target index. +When you are performing a [reindex]({{site.url}}{{site.baseurl}}/im-plugin/reindex/index/) operation from a source index, the new segments created in the target index will have the properties of the codec settings of the target index. ### Index rollups and transforms diff --git a/_im-plugin/index-context.md b/_im-plugin/index-other/index-context.md similarity index 97% rename from _im-plugin/index-context.md rename to _im-plugin/index-other/index-context.md index 5cbe6daa1b1..071ed742810 100644 --- a/_im-plugin/index-context.md +++ b/_im-plugin/index-other/index-context.md @@ -1,14 +1,16 @@ --- layout: default title: Index context -nav_order: 20 +parent: Other components +grand_parent: Managing indexes +nav_order: 60 redirect_from: - /opensearch/index-context/ --- # Index context -This is an experimental feature and is not recommended for use in a production environment. For updates on the progress the feature or if you want to leave feedback, join the discussion on the [OpenSearch forum](https://forum.opensearch.org/). +This is an experimental feature and is not recommended for use in a production environment. For updates on the progress the feature or if you want to leave feedback, join the discussion on the [OpenSearch forum](https://forum.opensearch.org/). {: .warning} Index context declares the use case for an index. Using the context information, OpenSearch applies a predetermined set of settings and mappings, which provides the following benefits: diff --git a/_im-plugin/index-sorting.md b/_im-plugin/index-other/index-sorting.md similarity index 98% rename from _im-plugin/index-sorting.md rename to _im-plugin/index-other/index-sorting.md index f8f76a48677..29728a07f06 100644 --- a/_im-plugin/index-sorting.md +++ b/_im-plugin/index-other/index-sorting.md @@ -1,9 +1,12 @@ --- layout: default title: Index sorting -nav_order: 25 +parent: Other components +grand_parent: Managing indexes +nav_order: 30 --- + # Index sorting OpenSearch allows you to configure how documents are organized within each segment at index creation time. By default, Lucene applies no sorting to documents. The `index.sort.*` settings specify how documents are organized within each segment. diff --git a/_im-plugin/index-other/index.md b/_im-plugin/index-other/index.md new file mode 100644 index 00000000000..dfc14c2e2f2 --- /dev/null +++ b/_im-plugin/index-other/index.md @@ -0,0 +1,39 @@ +--- +layout: default +title: Other components +nav_order: 130 +has_children: true +has_toc: false +--- + +# Other components and index management + +Several OpenSearch components and plugins affect indexing behavior and performance. See the following for more information. + +## The security plugin + +Manage role-based security for index management operations. See [Index management security]({{site.url}}{{site.baseurl}}/im-plugin/index-other/security/). + +## Index codecs + +Manage how an index's fields are compressed and stored. See [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/). + + +## Index sorting behavior + +Specify at creation how indexes are sorted on disk. See [Index sorting]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-sorting/). + + +## Similarity comparisons + +Control how document similarity is computed. See [Similarity]({{site.url}}{{site.baseurl}}/im-plugin/index-other/similarity/). + + +## Index search analyzers + +Refresh search analyzers in real time. See [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/index-other/refresh-analyzer/). + + +## Index context + +Declare profiles for indexing use cases (experimental). See [Index context]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-context/). diff --git a/_im-plugin/refresh-analyzer.md b/_im-plugin/index-other/refresh-analyzer.md similarity index 95% rename from _im-plugin/refresh-analyzer.md rename to _im-plugin/index-other/refresh-analyzer.md index a1dcff9fe86..6acb64bdc8a 100644 --- a/_im-plugin/refresh-analyzer.md +++ b/_im-plugin/index-other/refresh-analyzer.md @@ -1,7 +1,9 @@ --- layout: default title: Refresh search analyzer -nav_order: 35 +parent: Other components +grand_parent: Managing indexes +nav_order: 20 has_toc: false redirect_from: - /query-dsl/analyzers/refresh-analyzer/ diff --git a/_im-plugin/security.md b/_im-plugin/index-other/security.md similarity index 96% rename from _im-plugin/security.md rename to _im-plugin/index-other/security.md index 093afe36cfd..344726a50bf 100644 --- a/_im-plugin/security.md +++ b/_im-plugin/index-other/security.md @@ -1,7 +1,9 @@ --- layout: default title: Index management security -nav_order: 140 +parent: Other components +grand_parent: Managing indexes +nav_order: 10 has_children: false --- @@ -17,7 +19,7 @@ The Security plugin comes with one role that offers full access to index managem With security enabled, users not only need the correct index management permissions, but they also need permissions to execute actions to involved indexes. For example, if a user wants to use the REST API to attach a policy that executes a rollup job to an index named `system-logs`, they would need the permissions to attach a policy and execute a rollup job, as well as access to `system-logs`. -Finally, with the exceptions of Create Policy, Get Policy, and Delete Policy, users also need the `indices:admin/opensearch/ism/managedindex` permission to execute [ISM APIs]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/). +Finally, with the exceptions of Create Policy, Get Policy, and Delete Policy, users also need the `indices:admin/opensearch/ism/managedindex` permission to execute [ISM APIs]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism). ## (Advanced) Limit access by backend role diff --git a/_im-plugin/similarity.md b/_im-plugin/index-other/similarity.md similarity index 99% rename from _im-plugin/similarity.md rename to _im-plugin/index-other/similarity.md index b9c6c3fca0f..8670fc19246 100644 --- a/_im-plugin/similarity.md +++ b/_im-plugin/index-other/similarity.md @@ -1,12 +1,14 @@ --- layout: default title: Similarity -nav_order: 37 +parent: Other components +grand_parent: Managing indexes +nav_order: 50 --- # Similarity -A similarity defines how matching documents are scored and ranked during search operations. OpenSearch uses similarity algorithms to calculate relevance scores that determine the order of search results. +A similarity defines how matching documents are scored and ranked during search operations. OpenSearch uses similarity algorithms to calculate relevance scores that determine the order of search results. Each field can have its own similarity configuration, allowing fine-tuned control over how different types of content are scored. You can define custom similarity algorithms in your index settings at the index level. Once configured, you can apply these algorithms to specific fields using the [`similarity` mapping parameter]({{site.url}}{{site.baseurl}}/mappings/mapping-parameters/similarity/). diff --git a/_im-plugin/index-rollups/rollup-api.md b/_im-plugin/index-rollups/api-rollup.md similarity index 59% rename from _im-plugin/index-rollups/rollup-api.md rename to _im-plugin/index-rollups/api-rollup.md index cbe4b0f52ff..bc1ca7f984b 100644 --- a/_im-plugin/index-rollups/rollup-api.md +++ b/_im-plugin/index-rollups/api-rollup.md @@ -98,28 +98,28 @@ PUT _plugins/_rollup/jobs/{rollup_id}?if_seq_no=1&if_primary_term=1 // Update You can specify the following options. -Options | Description | Type | Required -:--- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--- |:--- | -`source_index` | The name of the detector. | String | Yes +Options | Description | Type | Required +:--- |:--- |:--- |:--- | +`source_index` | The name of the detector. | String | Yes `target_index` | Specify the target index that the rolled up data is ingested into. You can either create a new target index or use an existing index. The target index cannot be a combination of raw and rolled up data. This field supports dynamically generated index names like {% raw %}`rollup_{{ctx.source_index}}`{% endraw %}, where `source_index` cannot contain wildcards. | String | Yes -`target_index_settings` | Specify any [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/) to be applied to the target index created during the rollup. | Object | No -`schedule` | Schedule of the index rollup job which can be an interval or a cron expression. | Object | Yes -`schedule.interval` | Specify the frequency of execution of the rollup job. | Object | No -`schedule.interval.start_time` | Start time of the interval. | Timestamp | Yes -`schedule.interval.period` | Define the interval period. | String | Yes -`schedule.interval.unit` | Specify the time unit of the interval. | String | Yes -`schedule.interval.cron` | Optionally, specify a cron expression to define therollup frequency. | List | No -`schedule.interval.cron.expression` | Specify a Unix cron expression. | String | Yes -`schedule.interval.cron.timezone` | Specify timezones as defined by the IANA Time Zone Database. Defaults to UTC. | String | No -`description` | Optionally, describe the rollup job. | String | No -`enabled` | When true, the index rollup job is scheduled. Default is `true`. | Boolean | Yes -`continuous` | Specify whether or not the index rollup job continuously rolls up data forever or executes over the current dataset once and stops. Default is `false`. | Boolean | Yes -`error_notification` | Set up a Mustache message template for error notifications. For example, if an index rollup job fails, the system sends a message to a Slack channel. | Object | No -`page_size` | Specify the number of buckets to paginate at a time during rollup. | Number | Yes -`delay` | The number of milliseconds to delay execution of the index rollup job. | Long | No -`dimensions` | Specify aggregations to create dimensions for the roll up time window. Supported groups are `terms`, `histogram`, and `date_histogram`. For more information, see [Bucket Aggregations]({{site.url}}{{site.baseurl}}/opensearch/bucket-agg/). | Array | Yes +`target_index_settings` | Specify any [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/) to be applied to the target index created during the rollup. | Object | No +`schedule` | Schedule of the index rollup job which can be an interval or a cron expression. | Object | Yes +`schedule.interval` | Specify the frequency of execution of the rollup job. | Object | No +`schedule.interval.start_time` | Start time of the interval. | Timestamp | Yes +`schedule.interval.period` | Define the interval period. | String | Yes +`schedule.interval.unit` | Specify the time unit of the interval. | String | Yes +`schedule.interval.cron` | Optionally, specify a cron expression to define therollup frequency. | List | No +`schedule.interval.cron.expression` | Specify a Unix cron expression. | String | Yes +`schedule.interval.cron.timezone` | Specify timezones as defined by the IANA Time Zone Database. Defaults to UTC. | String | No +`description` | Optionally, describe the rollup job. | String | No +`enabled` | When true, the index rollup job is scheduled. Default is `true`. | Boolean | Yes +`continuous` | Specify whether or not the index rollup job continuously rolls up data forever or executes over the current dataset once and stops. Default is `false`. | Boolean | Yes +`error_notification` | Set up a Mustache message template for error notifications. For example, if an index rollup job fails, the system sends a message to a Slack channel. | Object | No +`page_size` | Specify the number of buckets to paginate at a time during rollup. | Number | Yes +`delay` | The number of milliseconds to delay execution of the index rollup job. | Long | No +`dimensions` | Specify aggregations to create dimensions for the roll up time window. Supported groups are `terms`, `histogram`, and `date_histogram`. For more information, see [Bucket Aggregations]({{site.url}}{{site.baseurl}}/opensearch/bucket-agg). | Array | Yes `routing_field` | The `source_field` of a `terms` dimension to use as the routing value for rolled up documents in the target index. When set, each rolled up document is indexed using the value of that dimension as its routing value. This ensures that searches specifying the same `routing` value are directed to the correct shard and can find the rolled up documents. If not set, rolled up documents are distributed across shards based on document ID, and searches that specify a `routing` value may not return matching documents. The value must match the `source_field` of one of the `terms` dimensions defined in `dimensions`. This setting is immutable and cannot be changed when updating an existing rollup job. Available in OpenSearch 3.7 and later. | String | No -`metrics` | Specify a list of objects that represent the fields and metrics that you want to calculate. Supported metrics are `sum`, `max`, `min`, `value_count` and `avg`. For more information, see [Metric Aggregations]({{site.url}}{{site.baseurl}}/opensearch/metric-agg/). | Array | No +`metrics` | Specify a list of objects that represent the fields and metrics that you want to calculate. Supported metrics are `sum`, `max`, `min`, `value_count` and `avg`. For more information, see [Metric Aggregations]({{site.url}}{{site.baseurl}}/opensearch/metric-agg). | Array | No #### Example response diff --git a/_im-plugin/index-rollups/dash-rollup.md b/_im-plugin/index-rollups/dash-rollup.md new file mode 100644 index 00000000000..239a784ddf6 --- /dev/null +++ b/_im-plugin/index-rollups/dash-rollup.md @@ -0,0 +1,94 @@ +--- +layout: default +title: Rollups in OpenSearch Dashboards +parent: Index rollups +nav_order: 30 +--- + + +# Creating a rollup job in OpenSearch Dashboards + +You can create rollup jobs that compress indexes to summarize historical data. This saves storage space and boosts aggregation performance. + +For example, say you collect CPU consumption data every five seconds and store it on a hot node. You can roll up this data to the average CPU consumption per day. + +For more information about rollup jobs, see [Index rollups]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/index/). + +To create a rollup job, follow these steps: + +1. In the **Index Management** panel, choose **Rollup jobs**. + +1. On the **Rollup jobs** page, choose the **Create rollup job** button. + +1. In the **Name** text box, enter a name for the new rollup job. + +1. (Optional) In the **Description** text box, enter a description. + +1. In the **Source index** combo box, enter or select the index pattern that you want to roll up. + +1. In the **Target index** combo box, select the index to receive the rollup results, or enter a name for a new target index. You can use embedded variables in the name. + +1. Choose **Next**. + +1. On the **Define aggregations and metrics** page, define the time aggregation in the **Time aggregation** panel by following these steps: + 1. Select the timestamp to aggregate from the **Timestamp field** drop-down. + 1. Choose an interval type, either **Fixed** or **Calendar**. Fixed intervals are of equal length. Calendar intervals are based on single calendar entities and can be unequal, for example as with months. + 1. In the **Interval** dropdown, select a number and interval (for Interval type) or a calendar interval (for Calendar type). + 1. In the **Timezone** drop-down, select the time zone that the timestamp is in. + +1. In the **Additional aggregation – _optional_** panel, specify one or more fields to aggregate by following these steps: + + 1. Select the **Add fields** button. + + 1. In the **Add fields** dialog, select one or more field names. + + 1. Select **Add**. + + 1. In the **Additional aggregations** table, for each numerical field select an aggregation (typically **Terms** or **Histogram**; you probably want **Histogram**) in the **Aggregation method** column. + + Keyword fields can only be aggregated by term. + {: .note} + + 1. For each numerical aggregation, enter an interval in the **Interval** column. This is the number of timestamp intervals per histogram bucket as defined in the **Time aggregation** panel. + + 1. (Optional) Arrange the order of the fields to optimize rollup performance. Aggregate fields with fewer bucket first, then more buckets. See [Index rollups]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/index/). + +1. (Optional) In the **Additional metrics _optional_** panel, select additional numeric fields to aggregate by following these steps: + + 1. In the **Additional metrics** panel, select **Add fields**. + + 1. Select one or more field names. + + 1. Select **Add**. + + 1. In the Additional metrics table, for each field, select one or more aggregations to save in the rollup. You can choose **Min**, **Max**, **Sum**, **Avg**, or **Value count**. + + Choosing **All** selects all aggregations for that field. + {: note} + + Use the **Disable all** and **Enable all** dropdowns to select and deselect an aggregation for all fields in the table. + {: .tip} + +1. In the lower right of the **Define aggregations and metrics** page, choose **Next**. + +1. On the **Specify schedule** page, set when the rollup job runs by following these steps: + + 1. To make the job run according to its execution schedule when not called by a policy, select **Enable job by default**. + + 1. To make the job run continuously, select **Continuous**. + + 1. From the **Rollup execution frequency** dropdown, select **Define by fixed interval** or **Define by cron expression**. + - If you selected fixed interval, select the interval from the number and units boxes in **Rollup interval**. + - If you selected cron expression, do the following: + 1. Enter a Unix cron expression in the **Define by cron expression** text box. + 1. Set the timezone under which the schedule is governed. + + 1. Enter a value in the page per execution combo box. This controls the tradeoff between execution performance and memory cost. + + 1. In the **Execution delay – _optional_** control enter the delay time in the number and units boxes. + +1. In the lower right of the **Define aggregations and metrics** page, choose **Next**. + +1. Review the settings for the rollup job. To edit the rollup job, select **Edit**. + +1. Choose **Create**. diff --git a/_im-plugin/index-rollups/index.md b/_im-plugin/index-rollups/index.md index bfd16e1f5fd..af07a1381e9 100644 --- a/_im-plugin/index-rollups/index.md +++ b/_im-plugin/index-rollups/index.md @@ -9,17 +9,17 @@ redirect_from: # Index rollups -Time-series data increases storage costs, strains cluster health, and slows down aggregations over time. Index rollup lets you periodically reduce data granularity by rolling up old data into summarized indexes. +Uncompressed time series data eventually increases storage costs, strains cluster health, and slows down aggregations. _Index rollup_ mitigates these effects by periodically compressing old data into summarized indexes with reduced granularity. -You pick the fields that interest you and use index rollup to create a new index with only those fields aggregated into coarser time buckets. You can store months or years of historical data at a fraction of the cost with the same query performance. +You pick the fields that interest you and use index rollup to create a new index with only those fields, aggregated into coarser time buckets. You can store months or years of historical data at a fraction of the cost with the same query performance. -For example, say you collect CPU consumption data every five seconds and store it on a hot node. Instead of moving older data to a read-only warm node, you can roll up or compress this data with only the average CPU consumption per day or with a 10% decrease in its interval every week. +For example, say you collect CPU consumption data every five seconds and store it on a hot node. Instead of moving older data to a read-only warm node, you can progressively compress this data with a 10% decrease in its interval every week. Or you could save only the average CPU consumption per day. You can use index rollup in three ways: 1. Use the Index Rollup API for an on-demand index rollup job that operates on an index that's not being actively ingested, such as a rolled-over index. For example, you can perform an index rollup operation to aggregate data collected at a 5-minute interval into a weekly average for trend analysis. -2. Use the OpenSearch Dashboards UI to create an index rollup job that runs on a defined schedule. You can also set it up to roll up your indexes as it’s being actively ingested. For example, you can continuously roll up Logstash indexes from a five second interval to a one hour interval. -3. Specify the index rollup job as an ISM action for complete index management. This allows you to roll up an index after a certain event such as a rollover, index age reaching a certain point, index becoming read-only, and so on. You can also have rollover and index rollup jobs running in sequence, where the rollover first moves the current index to a warm node and then the index rollup job creates a new index with the minimized data on the hot node. +2. Use the OpenSearch Dashboards UI to create an index rollup job that runs on a defined schedule. Or you can configure the job to roll up your indexes as they are being ingested. For example, you can continuously roll up Logstash indexes from a five second interval to a one hour interval. +3. Specify the index rollup job as an ISM action as a part of complete index management. This enables you to trigger a rollup after an event such as a rollover, index age reaching a certain point, index becoming read-only, and so on. You can also have rollover and index rollup jobs running in sequence, where the rollover first moves the current index to a warm node and then the index rollup job creates a new index with the minimized data on the hot node. ## Create an index rollup job @@ -1184,4 +1184,4 @@ The response contains two buckets, "Error" and "Success", and the document count ## Index codec considerations -For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#index-rollups-and-transforms). \ No newline at end of file +For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/#index-rollups-and-transforms). \ No newline at end of file diff --git a/_im-plugin/index-transforms/transforms-apis.md b/_im-plugin/index-transforms/api-transform.md similarity index 99% rename from _im-plugin/index-transforms/transforms-apis.md rename to _im-plugin/index-transforms/api-transform.md index 6af2543c77b..5e6ce21f06b 100644 --- a/_im-plugin/index-transforms/transforms-apis.md +++ b/_im-plugin/index-transforms/api-transform.md @@ -1,7 +1,7 @@ --- layout: default title: Transforms APIs -nav_order: 45 +nav_order: 10 parent: Index transforms has_toc: true --- diff --git a/_im-plugin/index-transforms/dash-transforms.md b/_im-plugin/index-transforms/dash-transforms.md new file mode 100644 index 00000000000..eb1ea38e38d --- /dev/null +++ b/_im-plugin/index-transforms/dash-transforms.md @@ -0,0 +1,80 @@ +--- +layout: default +title: Transforms in OpenSearch Dashboards +parent: Index transforms +nav_order: 30 +--- + + +# Creating a Transform Job in OpenSearch Dashboards + +You can create transform jobs that create realized views. These are a summarized view of selected fields that enable you to query or visualize the data. + +For example, say you have airline data that’s scattered across multiple fields and categories, and you want to view a summary of the data that’s organized by airline, quarter, and price. You can use a transform job to create a new index that’s organized by those categories. For more information about transform jobs, see [Index transforms]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms/index/). + +To create a transform job, follow these steps: + +1. In the **Index Management** panel, choose **Transform Jobs**. + +1. On the **Transform jobs** page, choose the **Create transform job** button. + +1. In the **Name** text box, enter a name for the new transform job. + +1. (Optional) In the **Description – _optional_** text box, enter a description. + +1. In the **Source index** combo box, enter or select the index that you want to transform. + +1. (Optional) Under the **Source index filter _optional_** heading, select the **Edit data filter** link to filter the source index data before applying the transform. To then edit the data filter, follow these steps: + + 1. In the Edit data filter dialog, enter the data filter using Query DSL. See [Query DSL]({{site.url}}{{site.baseurl}}/query-dsl/). + + For example, the following Query DSL expression filters all tickets costing less than $1000 from the sample flight data: + + ```json + { + "bool": { + "filter": [ + { "range": { "AvgTicketPrice": { "gte": "1000" }}} + ] + } + } + ``` + + 1. Select **Save**. + +1. In the **Target index** combo box, select the index to receive the transform results, or enter a name for a new target index. + +1. Choose **Next**. + +1. On the **Define transform** page, define the transform by following these steps: + 1. Select the **N columns hidden** link. + 1. In the fields drop-down, select the fields that you want in the transformed index. + + Select the **Hide all** link, then select the fields you want to add. + {: .tip} + + 1. For each of the fields in the **Original fields with sample data** table, follow these steps: + 1. Select the {::nomarkdown}plus icon{:/} icon to add a transform of the field. + 1. In the **Transform options** pop-up menu, select a grouping or aggregation. The aggregated field is added to the **Transformed fields preview based on sample data** table. + +1. Select **Next**. + +1. On the **Specify schedule** page, set when the transform job runs by following these steps: + + 1. To make the job run according to its execution schedule when not called by a policy, select **Enable job by default**. + + 1. To make the job run continuously, select **Continuous**. + + 1. In the **Transform execution interval **combo box, enter an interval value. + + 1. In the units drop-down, select **Minutes**, **Hours**, or **Days**. + + 1. (Optional) Select {::nomarkdown}expand icon{:/} (expand) **Advanced** to specify the number of pages per execution. + 1. Enter a value in the **Pages per execution** combo box. This controls the tradeoff between execution performance and memory cost. + +1. Select **Next**. + +1. On the **Review and Create** page, review the settings for the transform job. To edit a job, select the **Edit** button in the panel you want to edit. + +1. Choose **Create**. + diff --git a/_im-plugin/index-transforms/index.md b/_im-plugin/index-transforms/index.md index 6b88d27a017..de778167199 100644 --- a/_im-plugin/index-transforms/index.md +++ b/_im-plugin/index-transforms/index.md @@ -1,7 +1,7 @@ --- layout: default title: Index transforms -nav_order: 50 +nav_order: 70 has_children: true redirect_from: - /im-plugin/index-transforms/ @@ -155,4 +155,4 @@ GET finished_flight_job/_search ## Index codec considerations -For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#index-rollups-and-transforms). \ No newline at end of file +For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/#index-rollups-and-transforms). \ No newline at end of file diff --git a/_im-plugin/index.md b/_im-plugin/index.md index f44e7a83ce3..72bb761b301 100644 --- a/_im-plugin/index.md +++ b/_im-plugin/index.md @@ -1,288 +1,87 @@ --- layout: default title: Managing indexes -nav_order: 1 +nav_order: 10 has_children: false nav_exclude: true permalink: /im-plugin/ redirect_from: - /opensearch/index-data/ + - /opensearch/rest-api/index-apis/index/ - /im-plugin/index/ --- # Managing indexes -You index data using the OpenSearch REST API. Two APIs exist: the Index API and the `_bulk` API. +Indexes are the data structure at the heart of OpenSearch. If you're unfamiliar with OpenSearch data structures, see [Introduction to OpenSearch]({{site.url}}{{site.baseurl}}/getting-started/intro/). -For situations in which new data arrives incrementally (for example, customer orders from a small business), you might use the Index API to add documents individually as they arrive. For situations in which the data flow is less frequent (for example, weekly updates to a marketing website), you might prefer to generate a file and send it to the `_bulk` API. For large numbers of documents, lumping requests together and using the `_bulk` API offers superior performance. If your documents are exceptionally large, however, you might need to index them individually. +This section describes how to use the features of the Index plugin to manage indexes. The topics are organized in a manner roughly similar to their organization in the OpenSearch [Index APIs]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index/). -When indexing documents, the document `_id` must be 512 bytes or less in size. +Although the documentation roughly follows the API, most functionality is available in OpenSearch Dashboards, a web-based interface. In general, the API provides more fine-grained control and more options, but requires you to understand and write JSON-based queries. The OpenSearch Dashboards interface is more convenient, but in some cases is less flexible. +Index management does not include _[indexing]({{site.url}}{{site.baseurl}}/getting-started/index-data/)_, the process of populating indexes with data. It also does not include other data operations such as updating and querying documents. +{: .note} -## Introduction to indexing +## Index management concepts -Before you can search data, you must *index* it. Indexing is the method by which search engines organize data for fast retrieval. The resulting structure is called, fittingly, an index. +The following table provides links to conceptual information about index management. -In OpenSearch, the basic unit of data is a JSON *document*. Within an index, OpenSearch identifies each document using a unique ID. +To learn more about | Go to +:-- | :-- +Index operations | [Index operations]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/index/) +Index state management | [Index state management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/) +Index management concepts | Concepts: [Index management]({{site.url}}{{site.baseurl}}/getting-started/concepts/#index-management) -A request sent to the Index API appears as follows: +## Index management in OpenSearch Dashboards -```json -PUT {index}/_doc/{id} -{ "A JSON": "document" } -``` +The following table provides links to information about using OpenSearch Dashboards to manage indexes. -A request to the `_bulk` API looks a little different, because you specify the index and ID in the bulk data: +To learn more about | Go to +:-- | :-- +The OpenSearch Dashboards index management interface | [Index Management in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) +Core index operations: View, Create, Delete, Open, and Close indexes | [Core index operations in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops/) +Index management operations: Refresh, Flush, Clear cache, Force merge, Shrink, Split | [Managing indexes in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-manage/) +Index state management (ISM) | [Index state management with OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/ism/dash-ism/) +Adding data to indexes (_not an index management function_) | [Indexing documents]({{site.url}}{{site.baseurl}}/getting-started/communicate/#indexing-documents) +Index aliases: View, Create, Edit, Delete, Refresh, Flush, Clear cache, Rollover, and Force merge aliases | [Managing aliases with OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/aliases/dash-aliases/) +Index data streams: View, Create, Delete, Rollover, Refresh, Flush, and Clear cache of data streams | [Working with data streams with OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/) -```json -POST _bulk -{ "index": { "_index": "", "_id": "" } } -{ "A JSON": "document" } -``` -Bulk data must conform to a specific format, which requires a newline character (`\n`) at the end of every line, including the last line. This is the basic format: +## Index management using APIs -``` -Action and metadata\n -Optional document\n -Action and metadata\n -Optional document\n -``` +The following table provides links to information about using OpenSearch APIs to manage indexes. -The document is optional, because `delete` actions don't require a document. The other actions (`index`, `create`, and `update`) all require a document. If you specifically want the action to fail if the document already exists, use the `create` action instead of the `index` action. -{: .note } +To learn more about | Go to +:-- | :-- +The OpenSearch index APIs | [Index APIs]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index/) +Core index operations such as creating, opening, and closing indexes | [Core index APIs]({{site.url}}{{site.baseurl}}/api-reference/index-apis/core-index-apis/) +Index management operations, including rollovers, transforms, and reindexing | [Index operation APIs]({{site.url}}{{site.baseurl}}/api-reference/index-apis/index-operation-apis/) +Index state management (ISM) | [Index state management with the API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/) +Adding data to indexes (indexing — _not an index management function_) | [Ingest APIs]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/index/) +Index aliases: List, Add, Remove, Create, Update, Delete, and Check for aliases | [Alias APIs]({{site.url}}{{site.baseurl}}/api-reference/alias/) +Index alias examples | [Using the Aliases API]({{site.url}}{{site.baseurl}}/im-plugin/aliases/api-aliases/) +Index data streams: List, Add, Remove, Create, Update, Delete, and Check for aliases | [Alias APIs]({{site.url}}{{site.baseurl}}/api-reference/alias/) +Index data stream examples | [Using the Aliases API]({{site.url}}{{site.baseurl}}/im-plugin/aliases/api-aliases/) -To index bulk data using the `curl` command, navigate to the folder where you have your file saved and run the following command: -```json -curl -H "Content-Type: application/x-ndjson" -POST https://localhost:9200/data/_bulk -u 'admin:admin' --insecure --data-binary "@data.json" -``` +## Other index features -If any one of the actions in the `_bulk` API fail, OpenSearch continues to execute the other actions. Examine the `items` array in the response to figure out what went wrong. The entries in the `items` array are in the same order as the actions specified in the request. +The following table provides links to information about other features that support index management. -OpenSearch automatically creates an index when you add a document to an index that doesn't already exist. It also automatically generates an ID if you don't specify an ID in the request. This simple example automatically creates the movies index, indexes the document, and assigns it a unique ID: +To learn more about | Go to +:-| :-- +Index codecs | [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/) +Index context | [Index context]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-context/) +Index sorting | [Index sorting]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-sorting/) +Refreshing the search analyzer | [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/index-other/refresh-analyzer/) +Index security | [Index management security]({{site.url}}{{site.baseurl}}/im-plugin/index-other/security/) +Document similarity | [Similarity]({{site.url}}{{site.baseurl}}/im-plugin/index-other/similarity/) -```json -POST movies/_doc -{ "title": "Spirited Away" } -``` - -Automatic ID generation has a clear downside: because the indexing request didn't specify a document ID, you can't easily update the document at a later time. Also, if you run this request 10 times, OpenSearch indexes this document as 10 different documents with unique IDs. To specify an ID of 1, use the following request (note the use of PUT instead of POST): - -```json -PUT movies/_doc/1 -{ "title": "Spirited Away" } -``` - -Because you must specify an ID, if you run this command 10 times, you still have only one document indexed with the `_version` field incremented to 10. - -Indexes default to one primary shard and one replica. If you want to specify non-default settings, create the index before adding documents: - -```json -PUT more-movies -{ "settings": { "number_of_shards": 6, "number_of_replicas": 2 } } -``` - -## Naming restrictions for indexes - -OpenSearch indexes have the following naming restrictions: - -- All letters must be lowercase. -- Index names can't begin with underscores (`_`) or hyphens (`-`). -- Index names can't contain spaces, commas, or the following characters: - - `:`, `"`, `*`, `+`, `/`, `\`, `|`, `?`, `#`, `>`, or `<` - - - -## Read data - -After you index a document, you can retrieve it by sending a GET request to the same endpoint that you used for indexing: - -```json -GET movies/_doc/1 - -{ - "_index" : "movies", - "_type" : "_doc", - "_id" : "1", - "_version" : 1, - "_seq_no" : 0, - "_primary_term" : 1, - "found" : true, - "_source" : { - "title" : "Spirited Away" - } -} -``` - -You can see the document in the `_source` object. If the document is not found, the `found` key is `false` and the `_source` object is not part of the response. - -To retrieve multiple documents with a single command, use the `_mget` operation. -The format for retrieving multiple documents is similar to the `_bulk` operation, where you must specify the index and ID in the request body: - -```json -GET _mget -{ - "docs": [ - { - "_index": "", - "_id": "" - }, - { - "_index": "", - "_id": "" - } - ] -} -``` - -To only return specific fields in a document: - -```json -GET _mget -{ - "docs": [ - { - "_index": "", - "_id": "", - "_source": "field1" - }, - { - "_index": "", - "_id": "", - "_source": "field2" - } - ] -} -``` - -To check if a document exists: - -```json -HEAD movies/_doc/{doc-id} -``` - -If the document exists, you get back a `200 OK` response, and if it doesn't, you get back a `404 - Not Found` error. - -## Update data - -To update existing fields or to add new fields, send a POST request to the `_update` operation with your changes in a `doc` object: - -```json -POST movies/_update/1 -{ - "doc": { - "title": "Castle in the Sky", - "genre": ["Animation", "Fantasy"] - } -} -``` - -Note the updated `title` field and new `genre` field: - -```json -GET movies/_doc/1 - -{ - "_index" : "movies", - "_type" : "_doc", - "_id" : "1", - "_version" : 2, - "_seq_no" : 1, - "_primary_term" : 1, - "found" : true, - "_source" : { - "title" : "Castle in the Sky", - "genre" : [ - "Animation", - "Fantasy" - ] - } -} -``` - -The document also has an incremented `_version` field. Use this field to keep track of how many times a document is updated. - -POST requests make partial updates to documents. To altogether replace a document, use a PUT request: - -```json -PUT movies/_doc/1 -{ - "title": "Spirited Away" -} -``` - -The document with ID of 1 will contain only the `title` field, because the entire document will be replaced with the document indexed in this PUT request. - -Use the `upsert` object to conditionally update documents based on whether they already exist. Here, if the document exists, its `title` field changes to `Castle in the Sky`. If it doesn't, OpenSearch indexes the document in the `upsert` object. - -```json -POST movies/_update/2 -{ - "doc": { - "title": "Castle in the Sky" - }, - "upsert": { - "title": "Only Yesterday", - "genre": ["Animation", "Fantasy"], - "date": 1993 - } -} -``` - -### Example response - -```json -{ - "_index" : "movies", - "_type" : "_doc", - "_id" : "2", - "_version" : 2, - "result" : "updated", - "_shards" : { - "total" : 2, - "successful" : 1, - "failed" : 0 - }, - "_seq_no" : 3, - "_primary_term" : 1 -} -``` - -Each update operation for a document has a unique combination of the `_seq_no` and `_primary_term` values. - -OpenSearch first writes your updates to the primary shard and then sends this change to all the replica shards. An uncommon issue can occur if multiple users of your OpenSearch-based application make updates to existing documents in the same index. In this situation, another user can read and update a document from a replica before it receives your update from the primary shard. Your update operation then ends up updating an older version of the document. In the best case, you and the other user make the same changes, and the document remains accurate. In the worst case, the document now contains out-of-date information. - -To prevent this situation, use the `_seq_no` and `_primary_term` values in the request header: - -```json -POST movies/_update/2?if_seq_no=3&if_primary_term=1 -{ - "doc": { - "title": "Castle in the Sky", - "genre": ["Animation", "Fantasy"] - } -} -``` - -If the document is updated after we retrieved it, the `_seq_no` and `_primary_term` values are different and our update operation fails with a `409 — Conflict` error. - -When using the `_bulk` API, specify the `_seq_no` and `_primary_term` values within the action metadata. - -## Delete data - -To delete a document from an index, use a DELETE request: - -```json -DELETE movies/_doc/1 -``` - -The DELETE operation increments the `_version` field. If you add the document back to the same ID, the `_version` field increments again. This behavior occurs because OpenSearch deletes the document `_source`, but retains its metadata. ## Next steps -- The Index Management (IM) plugin lets you automate recurring index management activities and reduce storage costs. For more information, see [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/). +For instructions on how to add data to indexes, see [Ingest your data into OpenSearch]({{site.url}}{{site.baseurl}}/getting-started/ingest-data/). -- For instructions on how to reindex data, see [Reindex data]({{site.url}}{{site.baseurl}}/im-plugin/reindex-data/). +For information about the various ways to query data, including using Query DSL, SQL, or web-based graphical tools, see [Exploring Data]({{site.url}}{{site.baseurl}}/dashboards/#exploring-data). +For an introduction to the index management interface in OpenSearch Dashboards, see [Index Management in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/). diff --git a/_im-plugin/ism/api.md b/_im-plugin/ism/api-ism.md similarity index 99% rename from _im-plugin/ism/api.md rename to _im-plugin/ism/api-ism.md index 69ff05d5c3f..6ec2b86f4ee 100644 --- a/_im-plugin/ism/api.md +++ b/_im-plugin/ism/api-ism.md @@ -2,10 +2,10 @@ layout: default title: ISM API parent: Index State Management -nav_order: 20 +nav_order: 40 --- -# ISM API +# Index state management with the API Use the index state management operations to programmatically work with policies and managed indexes. @@ -177,7 +177,7 @@ Don't use the broad `*` wildcard, and instead add a prefix, such as `my-logs*`, Updates a policy. Use the `seq_no` and `primary_term` parameters to update an existing policy. If these numbers don't match the existing policy or the policy doesn't exist, ISM throws an error. -It's possible that the policy currently applied to your index isn't the most up-to-date policy available. To see what policy is currently applied to your index, see [Explain index]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#explain-index). To get the most up-to-date version of a policy, see [Get policy]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#get-policy). +It's possible that the policy currently applied to your index isn't the most up-to-date policy available. To see what policy is currently applied to your index, see [Explain index]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/#explain-index). To get the most up-to-date version of a policy, see [Get policy]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/#get-policy). #### Example request @@ -624,7 +624,7 @@ GET _plugins/_ism/explain/index_1 } ``` -Optionally, you can add the `show_policy` parameter to your request's path to get the policy that is currently applied to your index, which is useful for seeing whether the policy applied to your index is the latest one. To get the most up-to-date policy, see [Get Policy API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#get-policy). +Optionally, you can add the `show_policy` parameter to your request's path to get the policy that is currently applied to your index, which is useful for seeing whether the policy applied to your index is the latest one. To get the most up-to-date policy, see [Get Policy API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/#get-policy). #### Example request diff --git a/_im-plugin/ism/dash-ism.md b/_im-plugin/ism/dash-ism.md new file mode 100644 index 00000000000..1d2c557519c --- /dev/null +++ b/_im-plugin/ism/dash-ism.md @@ -0,0 +1,238 @@ +--- +layout: default +title: ISM with OpenSearch Dashboards +parent: Index State Management +grand_parent: Managing indexes +nav_order: 30 +--- + + +# Index state management with OpenSearch Dashboards + +Index state management policies (ISM policies) are configurations you can use to manage the lifecycles of indexes. For general information about ISM, see [Index state management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/). + +Policies are state machines. A policy consists of _states_, _actions_, and _transitions_, all of which you define. + +1. **States** are possible states of an index, including the default state for new indexes. For example, you might name your states `hot`, `warm`, or `delete`. For more information, see [States]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/#states). + +1. **Actions** are operations that you want the policy to perform, for example doing a rollover or force-merging, when an index enters a state. For more information, see [Actions]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/#actions). + +1. **Transitions** are conditions that must be met for an index to move into a new state. Transition conditions are typically based on age or file size. For example, "Move an index that is more than 8 weeks old to the `delete` state". For more information, see [Transitions]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/#transitions). + +There are no constraints on the configuration of your policies. An index can have any number of states, transition between any two states (or from a state into itself), and specify any number of actions in each state. + +For a complete description of policies, see [Policies]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/). + +For an example ISM template policy, see [Sample policy with ISM template for auto rollover]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies#sample-policy-with-ism-template-for-auto-rollover). + +Policies run periodically by default. This behavior is configurable. + + +## Viewing an index policy + +You can view existing index policies. + +To view an index policy, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, select **State management policies**. + +1. From the **State management policies** table, select the policy you want to look at. + + The page displays the Policy settings, ISM Templates, and States panels. + +1. (Optional) To view a state's transitions and actions, expand the state in the States panel by selecting the {::nomarkdown}expand icon{:/} (expand) icon next to the state's name. + + +## Creating an index policy + +To create an index policy, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. From the **Index Management** panel, choose **State management policies**. + +1. Select the **Create policy** button at the upper right of the Indexes panel. + + You can use the visual editor or JSON editor to create policies. We recommend using the visual editor to compose new policies. Use the JSON editor to paste an existing policy. + +1. In the **Configuration method** dialog, select **Visual editor**. + +1. Choose **Continue**. + +1. In the **Policy info** panel, name the policy. + 1. In the **Policy ID** field, enter an ID for the policy. The policy ID must be unique, and should be easy to recognize and remember (for example, `hot_cold_workflow`). + 1. (Optional) In the **Description** box, enter the purpose of the policy (for example, `Automatically transition indexes from hot to cold storage based on age`). + +1. (Optional) In the **Error notification – _(optional)_** panel, select a channel ID from the **Channel ID** drop-down. See [Error notifications]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies#error-notifications). + + If you're using auto rollovers in your policy, we recommend setting up error notifications. This notifies you of unexpectedly large indexes if rollovers fail. + {: .note} + +1. (Optional) In the **ISM templates – _optional_** panel, configure one or more templates to apply the policy automatically to future indexes. See [Creating ISM templates](#creating-ism-templates). + +1. Add states to the index policy. + + In the **States (0)** panel, select **Add state** to create and add states to the policy. See [Adding states to a policy](#adding-states-to-a-policy) for instructions. + + The number in parentheses after **States** shows the number of states currently defined in the policy. + {: .note} + +1. Select the **Create** button in the lower right of the **Create policy** page. + +## Adding states to a policy + +You create states, which include actions and transitions, as part of the index policy creation process. See [Creating an index policy](#creating-an-index-policy). + +To add a state, go to the **States** panel of the **Create policy** page, then follow these steps: + +1. In the **State name** field, enter a descriptive name for the state (for example, `hot`, `warm`, `cold`, or `delete`). + +1. Under **Order**, define the state's position. (Skip this step when defining the first state of policy.) + 1. Select **Add before** or **Add after**. + 1. Select the state that this position is relative to. + +1. (Optional) Define one or more actions for the state. + 1. Select **Add action**. + 1. Select an action from the **Action type** drop-down. + 1. If applicable, enter parameters for the action. For example, for the Snapshot action you must supply a repository and a name for the snapshot. + 1. (Optional) Define timeout and retry settings. + 1. In the **Timeout** box, enter the timeout period for the action; for example, `5h`. + 1. In the **Retry count** combo box, enter the number of retries before the action fails. + 1. In the **Retry backoff** drop-down, select a retry backoff policy. + 1. In the **Retry delay** box, enter the delay between retries; for example, `1d'. + 1. Select the **Add action** button. + + For more information about available actions, see [Actions](https://opensearch.org/docs/im-plugin/ism/policies/#actions). + +1. (Optional) In the **Transitions** section, define and add one or more transitions from this state. + + To add a transition, follow these steps: + + 1. In the **State** dialog, choose **+ Add Transition**. + + 1. In the **Destination state** combo box, enter or select the _target_, the state to transition to. + + A state can transition to itself. To define a state as its own target, enter the state name since it's not in the selection list yet. + {: .tip} + + 1. In the **Condition** drop-down, select the trigger condition for the transition. + 1. Enter any parameters required for the condition. For example, the _Minimum Doc Count_ condition requires that you specify the minimum number of documents to trigger the transition. + +1. Choose the **Add transition** button to add the transition. + +For more information about transitions, see [Transitions](https://opensearch.org/docs/im-plugin/ism/policies/#transitions). + +1. To save the completed state, choose the **Save state** button. + + +## Setting the initial state + +To specify which state your policy begins in, follow these steps: + +1. In States panel on the Create policy page, Select a state from the **Initial state** dropdown. + + +## Changing the default policy job interval + +See [Settings]({{site.url}}{{site.baseurl}}/im-plugin/ism/settings/). + + +## Creating ISM templates + +An ISM template is an index pattern that ISM uses to apply an index policy to new indexes. + +An ISM template has a priority. If more than one ISM template matches an index pattern, ISM uses the priority value to determine which template to apply. + +To create one or more ISM templates, follow these steps on the **Create policy** page: + + 1. In the **ISM templates - _optional_** panel, select **Add template**. + + 1. In the **Index patterns** box, enter an index pattern. + + For example, if you specify a template of `sample-index-*`, the ISM plugin automatically applies this policy to any indexes whose names start with `sample-index-`. + + An index pattern cannot contain any of the following characters: `:`, `"`, `+`, `/`, `\`, `|`, `?`, `#`, `>`, or `<`. + {: .note} + + 1. In the Priority combo box, enter or select a priority for the index pattern. + + If more than one ISM template that matches an index pattern, ISM uses the priority value to determine which template to apply. + + 1. (Optional) Repeat the previous steps to add more templates. + + For more information about ISM templates, see [Index templates](https://opensearch.org/docs/im-plugin/ism/index/). + + +## Applying a policy + +You can apply a policy to one or more indexes so that the state of the indexes is governed by the policy. + +To attach a policy to one or more indexes, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, choose **Indexes**. + +1. Select the checkboxes of one or more indexes to which you want to apply a policy. + +1. Choose **Actions**. + +1. From the **Actions** dropdown, select **Apply policy**. + +1. In the **Apply policy** dialog, select a policy from the **Policy ID** drop-down. + +1. Choose the **Apply** button. + + After you apply a policy to an index, ISM creates a job that runs every 5 minutes by default to perform policy actions, check conditions, and transition the index into different states. To change the default time interval for this job, see [Changing the default policy job interval](#changing-the-default-policy-job-interval). + + Policy jobs do not run if the cluster state is red. + {: .important} + +## Editing an index policy + +You can edit an index policy after it is created. The modifications are in force the next time the index policy is applied. + +To edit an index policy, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. In the **Index Management** panel, select **State management policies**. + +1. From the **State management policies** table, select the policy you want to edit. + +1. Select the **Edit** button in the upper right of the policy's viewing page. + +1. In the **Configuration method** dialog, choose **Visual editor**. + +1. Enter or change the policy features as described in [Creating an index policy](#creating-an-index-policy). + + You can edit any feature except the policy ID. + + +## Deleting index policies + +You can delete index policies. A deleted policy immediately ceases to apply to any indexes. + +Deleted polices are not recoverable. +{: .warning} + +To delete one or more index policies, follow these steps: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + +1. From the **Index Management** panel, select State management policies. + +1. In the **State management policies** table, select the checkboxes for the policies you want to delete. + +1. Choose **Delete** from the upper right of the **Index Management** page. + +1. In the Delete confirmation dialog, choose the **Delete** button. + + +## Next steps + +For more information about creating and setting up indexes, see [Core index operations]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/index/). + +For more information about managing indexes without policies, see [Managing indexes]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-manage). diff --git a/_im-plugin/ism/error-prevention/api.md b/_im-plugin/ism/error-prevention/api.md index d9a784838b6..b2a42e4bc96 100644 --- a/_im-plugin/ism/error-prevention/api.md +++ b/_im-plugin/ism/error-prevention/api.md @@ -3,7 +3,7 @@ layout: default title: ISM Error Prevention API parent: ISM Error Prevention grand_parent: Index State Management -nav_order: 10 +nav_order: 20 --- # ISM Error Prevention API diff --git a/_im-plugin/ism/error-prevention/index.md b/_im-plugin/ism/error-prevention/index.md index 5283c49adf5..e27b1116361 100644 --- a/_im-plugin/ism/error-prevention/index.md +++ b/_im-plugin/ism/error-prevention/index.md @@ -1,7 +1,7 @@ --- layout: default title: ISM Error Prevention -nav_order: 90 +nav_order: 120 has_children: true has_toc: false redirect_from: @@ -10,7 +10,7 @@ redirect_from: # ISM error prevention -Error prevention validates Index State Management (ISM) actions before they are performed in order to prevent actions from failing. It also outputs additional information from the action validation results in the response of the [Index Explain API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#explain-index). Validation rules and troubleshooting of each action are listed in the following sections. +Error prevention validates Index State Management (ISM) actions before they are performed in order to prevent actions from failing. It also outputs additional information from the action validation results in the response of the [Index Explain API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/#explain-index). Validation rules and troubleshooting of each action are listed in the following sections. --- diff --git a/_im-plugin/ism/error-prevention/resolutions.md b/_im-plugin/ism/error-prevention/resolutions.md index 04af4ad43f0..9dd331a8185 100644 --- a/_im-plugin/ism/error-prevention/resolutions.md +++ b/_im-plugin/ism/error-prevention/resolutions.md @@ -3,7 +3,7 @@ layout: default title: ISM error prevention resolutions parent: ISM Error Prevention grand_parent: Index State Management -nav_order: 5 +nav_order: 10 --- # ISM error prevention resolutions @@ -101,7 +101,7 @@ PUT {target_index}/_settings ## This index has already been rolled over successfully -Remove the [rollover policy from the index]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#remove-policy-from-index) to prevent this error from reoccurring. +Remove the [rollover policy from the index]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/#remove-policy-from-index) to prevent this error from reoccurring. ## The rollover policy misses rollover_alias index setting diff --git a/_im-plugin/ism/index.md b/_im-plugin/ism/index.md index 8cbc1bce506..4f692f2628c 100644 --- a/_im-plugin/ism/index.md +++ b/_im-plugin/ism/index.md @@ -12,11 +12,11 @@ has_toc: false If you analyze time-series data, you likely prioritize new data over old data. You might periodically perform certain operations on older indexes, such as reducing replica count or deleting them. -Index State Management (ISM) lets you automate these periodic, administrative operations by triggering them based on changes in the index age, index size, or number of documents. Using the ISM plugin, you can define *policies* that automatically handle index rollovers or deletions to fit your use case. +Index State Management (ISM) automates these periodic, administrative operations by triggering them based on changes in the index age, index size, or number of documents. Using the ISM plugin, you can define *policies* that automatically handle index rollovers or deletions to fit your use case. -For example, you can define a policy that moves your index into a `read_only` state after 30 days and then deletes it after a set period of 90 days. You can also set up the policy to send you a notification message when the index is deleted. +For example, you can define a policy that moves your index into a `read_only` state after 30 days and then deletes it after 90 days. You can also set up the policy to send you a notification message when the index is deleted. -You might want to perform an index rollover after a certain amount of time or run a `force_merge` operation on an index during off-peak hours to improve search performance during peak hours. +As another example, you might want to perform an index rollover after a certain amount of time or run a `force_merge` operation on an index during off-peak hours to improve search performance during peak hours. To use the ISM plugin, your user role needs to be mapped to the `all_access` role that gives you full access to the cluster. To learn more, see [Users and roles]({{site.url}}{{site.baseurl}}/security/access-control/users-roles/). {: .note } @@ -27,7 +27,7 @@ To get started, choose **Index Management** in OpenSearch Dashboards. ### Step 1: Set up policies -A policy is a set of rules that describes how an index should be managed. For information about creating a policy, see [Policies]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/). +A policy is a set of rules that describes how an index should be managed. For information about policies and their components, see [Policies]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/). You can use the visual editor or JSON editor to create policies. Compared to the JSON editor, the visual editor offers a more structured way of defining policies by separating the process into creating error notifications, defining ISM templates, and adding states. We recommend using the visual editor if you want to see predefined fields, such as which actions you can assign to a state or under what conditions a state can transition into a destination state. diff --git a/_im-plugin/ism/managedindexes.md b/_im-plugin/ism/managedindexes.md index a0dc52c7297..1207fcbcc7d 100644 --- a/_im-plugin/ism/managedindexes.md +++ b/_im-plugin/ism/managedindexes.md @@ -1,18 +1,39 @@ --- layout: default title: Managed indexes -nav_order: 3 -parent: Index State Management +nav_order: 10 +parent: ISM API +grand_parent: Index State Management has_children: false redirect_from: - /im-plugin/ism/managedindices/ --- + # Managed indexes You can change or update a policy using the managed index operations. -This table lists the fields of managed index operations. + +## Change policy + +You can change any managed index policy, with some caveats. ISM has constraints in place to make sure that policy changes don't break indexes. + +If an index is stuck in its current state, never proceeding, and you want to update its policy immediately, make sure that the new policy includes the same state---same name, same actions, same order---as the old policy. In this case, even if the policy is in the middle of executing an action, ISM applies the new policy. + +If you update the policy without including an identical state, ISM updates the policy only after all actions in the current state finish executing. Alternately, you can choose a specific state in your old policy after which you want the new policy to take effect. + +To change a policy using OpenSearch Dashboards, do the following: + +- Under **Index Management**, choose the indexes that you want to attach the new policy to. +- To attach the new policy to indexes in specific states, choose **Choose state filters**, and then choose those states. +- Under **Choose New Policy**, choose the new policy. +- To start the new policy for indexes in the current state, choose **Keep indices in their current state after the policy takes effect**. +- To start the new policy in a specific state, choose **Start from a chosen state after changing policies**, and then choose the default start state in your new policy. + +## Change policy parameters + +This table lists the parameters of the Change Policy operation. Parameter | Description | Type | Required | Read Only :--- | :--- |:--- |:--- | @@ -58,18 +79,3 @@ The following example shows a managed index policy: } ``` -## Change policy - -You can change any managed index policy, but ISM has a few constraints in place to make sure that policy changes don't break indexes. - -If an index is stuck in its current state, never proceeding, and you want to update its policy immediately, make sure that the new policy includes the same state---same name, same actions, same order---as the old policy. In this case, even if the policy is in the middle of executing an action, ISM applies the new policy. - -If you update the policy without including an identical state, ISM updates the policy only after all actions in the current state finish executing. Alternately, you can choose a specific state in your old policy after which you want the new policy to take effect. - -To change a policy using OpenSearch Dashboards, do the following: - -- Under **Index Management**, choose the indexes that you want to attach the new policy to. -- To attach the new policy to indexes in specific states, choose **Choose state filters**, and then choose those states. -- Under **Choose New Policy**, choose the new policy. -- To start the new policy for indexes in the current state, choose **Keep indices in their current state after the policy takes effect**. -- To start the new policy in a specific state, choose **Start from a chosen state after changing policies**, and then choose the default start state in your new policy. diff --git a/_im-plugin/ism/policies-examples.md b/_im-plugin/ism/policies-examples.md new file mode 100644 index 00000000000..3599505f70e --- /dev/null +++ b/_im-plugin/ism/policies-examples.md @@ -0,0 +1,315 @@ +--- +layout: default +title: Policy examples +nav_order: 20 +parent: Policies +grand_parent: Index State Management +has_children: false +--- + +# Policy examples + +This page contains several examples of complete policies in JSON format. + +For a description of the components of a policy, see [Policies]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/). + +To find out how to manage policies from OpenSearch Dashboards, see [ISM with OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/ism/dash-ism/). + +To find out how to manage policies using APIs, see [ISM API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/). + + +## Sample policy with ISM template for auto rollover + +The following sample template policy is for a rollover use case. + +If you want to skip rollovers for an index, set `index.plugins.index_state_management.rollover_skip` to `true` in the settings of that index. + +1. Create a policy with an `ism_template` field: + + ```json + PUT _plugins/_ism/policies/rollover_policy + { + "policy": { + "description": "Example rollover policy.", + "default_state": "rollover", + "states": [ + { + "name": "rollover", + "actions": [ + { + "rollover": { + "min_doc_count": 1 + } + } + ], + "transitions": [] + } + ], + "ism_template": { + "index_patterns": ["log*"], + "priority": 100 + } + } + } + ``` + {% include copy-curl.html %} + + You need to specify the `index_patterns` field. If you don't specify a value for `priority`, it defaults to 0. + +2. Set up a template with the `rollover_alias` as `log` : + + ```json + PUT _index_template/ism_rollover + { + "index_patterns": ["log*"], + "template": { + "settings": { + "plugins.index_state_management.rollover_alias": "log" + } + } + } + ``` + {% include copy-curl.html %} + +3. Create an index with the `log` alias: + + ```json + PUT log-000001 + { + "aliases": { + "log": { + "is_write_index": true + } + } + } + ``` + {% include copy-curl.html %} + +4. Index a document to trigger the rollover condition: + + ```json + POST log/_doc + { + "message": "dummy" + } + ``` + {% include copy-curl.html %} + +5. Verify if the policy is attached to the `log-000001` index: + + ```json + GET _plugins/_ism/explain/log-000001?pretty + ``` + {% include copy-curl.html %} + +## Example policy with ISM templates for the alias action + +The following example policy is for an alias action use case. + +In the following example, the first job will trigger the rollover action, and a new index will be created. Next, another document is added to the two indexes. The new job will then cause the second index to point to the log alias, and the older index will be removed due to the alias action. + +First, create an ISM policy: + +```json +PUT /_plugins/_ism/policies/rollover_policy?pretty +{ + "policy": { + "description": "Example rollover policy.", + "default_state": "rollover", + "states": [ + { + "name": "rollover", + "actions": [ + { + "rollover": { + "min_doc_count": 1 + } + } + ], + "transitions": [{ + "state_name": "alias", + "conditions": { + "min_doc_count": "2" + } + }] + }, + { + "name": "alias", + "actions": [ + { + "alias": { + "actions": [ + { + "remove": { + "alias": "log" + } + } + ] + } + } + ] + } + ], + "ism_template": { + "index_patterns": ["log*"], + "priority": 100 + } + } +} +``` +{% include copy-curl.html %} + +Next, create an index template on which to enable the policy: + +```json +PUT /_index_template/ism_rollover? +{ + "index_patterns": ["log*"], + "template": { + "settings": { + "plugins.index_state_management.rollover_alias": "log" + } + } +} +``` +{% include copy-curl.html %} + +Next, change the cluster settings to trigger jobs every minute: + +```json +PUT /_cluster/settings?pretty=true +{ + "persistent" : { + "plugins.index_state_management.job_interval" : 1 + } +} +``` +{% include copy-curl.html %} + +Next, create a new index: + +```json +PUT /log-000001 +{ + "aliases": { + "log": { + "is_write_index": true + } + } +} +``` +{% include copy-curl.html %} + +Finally, add a document to the index to trigger the job: + +```json +POST /log-000001/_doc +{ + "message": "dummy" +} +``` +{% include copy-curl.html %} + +You can verify these steps using the Alias and Index API: + +```json +GET /_cat/indices?pretty +``` +{% include copy-curl.html %} + +```json +GET /_cat/aliases?pretty +``` +{% include copy-curl.html %} + +Note: The `index` and `remove_index` parameters are not allowed with alias action policies. Only the `add` and `remove` alias action parameters are allowed. +{: .warning } + +## Example policy + +The following example policy implements a `hot`, `warm`, and `delete` workflow. You can use this policy as a template to prioritize resources to your indexes based on their levels of activity. + +In this case, an index is initially in a `hot` state. After 7 days, it changes to a `warm` state, where the number of replicas is reduced to 1 and the indexes are moved to nodes with the `warm` attribute. + +After 30 days, the policy moves this index into a `delete` state. The service sends a notification to a Chime room that the index is being deleted, and then permanently deletes it. + +```json +{ + "policy": { + "description": "hot warm delete workflow", + "default_state": "hot", + "schema_version": 1, + "states": [ + { + "name": "hot", + "actions": [ + { + "rollover": { + "min_index_age": "7d", + "min_primary_shard_size": "30gb" + } + } + ], + "transitions": [ + { + "state_name": "warm" + } + ] + }, + { + "name": "warm", + "actions": [ + { + "replica_count": { + "number_of_replicas": 1 + } + }, + { + "allocation": { + "require": { + "temp": "warm" + } + } + } + ], + "transitions": [ + { + "state_name": "delete", + "conditions": { + "min_index_age": "30d" + } + } + ] + }, + { + "name": "delete", + "actions": [ + { + "notification": { + "destination": { + "chime": { + "url": "" + } + }, + "message_template": { + "source": "The index {% raw %}{{ctx.index}}{% endraw %} is being deleted" + } + } + }, + { + "delete": {} + } + ] + } + ], + "ism_template": { + "index_patterns": ["log*"], + "priority": 100 + } + } +} +``` + +This diagram shows the `states`, `transitions`, and `actions` of the preceding policy as a finite-state machine. For more information about finite-state machines, see [Wikipedia](https://en.wikipedia.org/wiki/Finite-state_machine). + +![Policy State Machine]({{site.url}}{{site.baseurl}}/images/ism.png) diff --git a/_im-plugin/ism/policies-operations.md b/_im-plugin/ism/policies-operations.md new file mode 100644 index 00000000000..7edc9aaf75b --- /dev/null +++ b/_im-plugin/ism/policies-operations.md @@ -0,0 +1,675 @@ +--- +layout: default +title: ISM supported operations +nav_order: 10 +parent: Policies +grand_parent: Index State Management +has_children: false +--- + + +# ISM supported operations + +ISM supports the following operations: + +- [Force merge](#force-merge) +- [Read only](#read-only) +- [Read write](#read-write) +- [Replica count](#replica-count) +- [Shrink](#shrink) +- [Close](#close) +- [Open](#open) +- [Delete](#delete) +- [Rollover](#rollover) +- [Notification](#notification) +- [Snapshot](#snapshot) +- [Convert index to remote](#convert-index-to-remote) +- [Index priority](#index-priority) +- [Allocation](#allocation) +- [Rollup](#rollup) +- [Stop replication](#stop-replication) +- [Search only](#search-only) + +## Force merge + +Reduces the number of Lucene segments by merging the segments of individual shards. This operation attempts to set the index to a `read-only` state before starting the merging process. + +Parameter | Description | Type | Required +:--- | :--- |:--- |:--- | +`max_num_segments` | The number of segments to reduce the shard to. | Integer | Yes + +```json +{ + "force_merge": { + "max_num_segments": 1 + } +} +``` + +## Read only + +Sets a managed index to be read only. + +```json +{ + "read_only": {} +} +``` + +Set the index setting `index.blocks.write` to `true` for a managed index. ***Note:** this block does not prevent the index from refreshing. + +## Read write + +Sets a managed index to be writeable. + +```json +{ + "read_write": {} +} +``` + +## Replica count + +Sets the number of replicas to assign to an index. + +Parameter | Description | Type | Required +:--- | :--- |:--- |:--- | +`number_of_replicas` | Defines the number of replicas to assign to an index. | Integer | Yes + +```json +{ + "replica_count": { + "number_of_replicas": 2 + } +} +``` + +For information about setting replicas, see [Primary and replica shards]({{site.url}}{{site.baseurl}}/intro/#primary-and-replica-shards). + +## Shrink + +Allows you to reduce the number of primary shards in your indexes. With this action, you can specify: + +- The number of primary shards that the target index should contain. +- A max shard size for the primary shards in the target index. +- Specify a percentage to shrink the number of primary shards in the target index. + +```json +"shrink": { + "num_new_shards": 1, + "target_index_name_template": { + "source": "{{ctx.index}}_shrunken" + }, + "aliases": [ + { + "my-alias": {} + } + ], + "switch_aliases": true, + "force_unsafe": false +} +``` + +Parameter | Description | Type | Example | Required +:--- | :--- |:--- |:--- | +`num_new_shards` | The maximum number of primary shards in the shrunken index. | Integer | `5` | Yes. It, however, cannot be used with `max_shard_size` or `percentage_of_source_shards`. +`max_shard_size` | The maximum size in bytes of a shard for the target index. | Keyword | `5gb` | Yes, however, it cannot be used with `num_new_shards` or `percentage_of_source_shards`. +`percentage_of_source_shards` | Percentage of the number of original primary shards to shrink. This parameter indicates the minimum percentage to use when shrinking the number of primary shards. Must be between 0.0 and 1.0, exclusive. | Percentage | `0.5` | Yes, however it cannot be used with `max_shard_size` or `num_new_shards` +`target_index_name_template` | The name of the shrunken index. Accepts strings and the Mustache variables `{% raw %}{{ctx.index}}{% endraw %}` and `{% raw %}{{ctx.indexUuid}}{% endraw %}`. | String or Mustache template | `{"source": "{% raw %}{{ctx.index}}_shrunken"}{% endraw %}` | No +`aliases` | Aliases to add to the new index. | Object | `myalias` | No. It must be an array of alias objects. +`switch_aliases` | If `true`, copies the aliases from the source index to the target index. If there is a name conflict with an alias from the `aliases` field, the alias in the `aliases` field is used instead of the name. | Boolean | `true` | No. The default implicit value is `false`, which means no aliases are copied by default. +`force_unsafe` | If `true`, shrinks the index even if it has no replicas. | Boolean | `false` | No + +If you want to add `aliases` to the action, the parameter must include an array of [alias objects]({{site.url}}{{site.baseurl}}/api-reference/alias/). For example, + +```json +"aliases": [ + { + "my-alias": {} + }, + { + "my-second-alias": { + "is_write_index": false, + "filter": { + "multi_match": { + "query": "QUEEN", + "fields": ["speaker", "text_entry"] + } + }, + "index_routing" : "1", + "search_routing" : "1" + } + }, +] +``` + +## Close + +Closes the managed index. + +```json +{ + "close": {} +} +``` + +Closed indexes remain on disk, but consume no CPU or memory. You can't read from, write to, or search closed indexes. + +Closing an index is a good option if you need to retain data for longer than you need to actively search it and have sufficient disk space on your data nodes. If you need to search the data again, reopening a closed index is simpler than restoring an index from a snapshot. + +## Open + +Opens a managed index. + +```json +{ + "open": {} +} +``` + +## Delete + +Deletes a managed index. + +```json +{ + "delete": {} +} +``` + +## Rollover + +Rolls an alias over to a new index when the managed index meets one of the rollover conditions. + +

+ +> **IMPORTANT** +> +>ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will be performed if the value **has reached** or _has exceeded_ the configured limit **when the check is performed**. For example, with `min_size` configured to a value of 100 GiB, ISM might check the index at 99 GiB and not perform the rollover. However, if the index has grown past the limit by the next check (for example, to 105 GiB), the operation is performed. +{: .important} + +If you need to skip the rollover action, you can set the index setting `index.plugins.index_state_management.rollover_skip` to `true`. For example, if you receive the error message "Missing alias or not the write index...", you can set the `index.plugins.index_state_management.rollover_skip` parameter to `true` and retry to skip rollover action. + +The index format must match the pattern: `^.*-\d+$`. For example, `(logs-000001)`. +Set `index.plugins.index_state_management.rollover_alias` as the alias to rollover. + +Parameter | Description | Type | Example | Required +:--- | :--- |:--- |:--- | +`min_size` | The minimum size of the total primary shard storage (not counting replicas) required to roll over the index. For example, if you set `min_size` to 100 GiB and your index has 5 primary shards and 5 replica shards of 20 GiB each, the total size of all primary shards is 100 GiB, so the rollover occurs. See [**Important** note](#important-note). | String | `20gb` or `5mb` | No +`min_primary_shard_size` | The minimum storage size of a **single primary shard** required to roll over the index. For example, if you set `min_primary_shard_size` to 30 GiB and **one of** the primary shards in the index has a size greater than the condition, the rollover occurs. See [**Important** note](#important-note). | String | `20gb` or `5mb` | No +`min_doc_count` | The minimum number of documents required to roll over the index. See [**Important** note](#important-note). | Integer | `2000000` | No +`min_index_age` | The minimum age required to roll over the index. Index age is the time between its creation and the present. Supported units are `d` (days), `h` (hours), `m` (minutes), `s` (seconds), `ms` (milliseconds), and `micros` (microseconds). See [**Important** note](#important-note). | String | `5d` or `7h` | No +`copy_alias` | Controls whether to copy over all aliases from the current index to a newly created index. Defaults to `false`. | `boolean` | `true` or `false` | No + +```json +{ + "rollover": { + "min_size": "50gb" + } +} +``` + +```json +{ + "rollover": { + "min_primary_shard_size": "30gb" + } +} +``` + +```json +{ + "rollover": { + "min_doc_count": 100000000 + } +} +``` + +```json +{ + "rollover": { + "min_index_age": "30d" + } +} +``` + +## Notification + +Sends you a notification. + +Parameter | Description | Type | Required +:--- | :--- |:--- |:--- | +`destination` | The destination URL. | `Slack, Amazon Chime, or webhook URL` | Yes +`message_template` | The text of the message. You can add variables to your messages using [Mustache templates](https://mustache.github.io/mustache.5.html). | `object` | Yes + +The destination system **must** return a response otherwise the notification operation throws an error. + +### Example 1: Chime notification + +```json +{ + "notification": { + "destination": { + "chime": { + "url": "" + } + }, + "message_template": { + "source": "the index is {% raw %}{{ctx.index}}{% endraw %}" + } + } +} +``` + +### Example 2: Custom webhook notification + +```json +{ + "notification": { + "destination": { + "custom_webhook": { + "url": "https://" + } + }, + "message_template": { + "source": "the index is {% raw %}{{ctx.index}}{% endraw %}" + } + } +} +``` + +### Example 3: Slack notification + +```json +{ + "notification": { + "destination": { + "slack": { + "url": "https://hooks.slack.com/services/xxx/xxxxxx" + } + }, + "message_template": { + "source": "the index is {% raw %}{{ctx.index}}{% endraw %}" + } + } +} +``` + +You can use `ctx` variables in your message to represent a number of policy parameters based on the past executions of your policy. For example, if your policy has a rollover action, you can use `{% raw %}{{ctx.action.name}}{% endraw %}` in your message to represent the name of the rollover. + +The following `ctx` variable options are available for every policy: + +### Guaranteed variables + +Parameter | Description | Type +:--- | :--- |:--- |:--- | +`index` | The name of the index. | String +`index_uuid` | The UUID of the index. | String +`policy_id` | The name of the policy. | String + +## Snapshot + +Back up your cluster’s indexes and state. For more information about snapshots, see [Take and restore snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore/). + +The `snapshot` operation has the following parameters. + +Parameter | Description | Type | Required | Default +:--- | :--- |:--- |:--- | +`repository` | The repository name that you register through the native snapshot API operations. | String | Yes | - +`snapshot` | The name of the snapshot. Accepts strings and the Mustache variables `{% raw %}{{ctx.index}}{% endraw %}` and `{% raw %}{{ctx.indexUuid}}{% endraw %}`. If the Mustache variables are invalid, then the snapshot name defaults to the index's name. | String or Mustache template | Yes | - + +```json +{ + "snapshot": { + "repository": "my_backup", + "snapshot": "{% raw %}{{ctx.indexUuid}}{% endraw %}" + } +} +``` + +## Convert index to remote + +Converts an existing index into a searchable snapshot by restoring it from a remote snapshot repository. This action reduces storage costs by moving infrequently accessed data to remote storage while keeping it searchable. After the restore request is accepted, the original index is automatically deleted, ensuring that only the remote snapshot-backed index remains. + +The `convert_index_to_remote` operation has the following parameters. + +Parameter | Description | Type | Required | Default +:--- | :--- |:--- |:--- | +`repository` | The repository name registered through the native snapshot API operations. Must be a remote repository (for example, S3, Azure, or GCS). | String | Yes | N/A +`snapshot` | The name of the snapshot created by the snapshot action. | String | Yes | N/A +`include_aliases` | Whether to include index aliases during the restore operation. If `true`, all aliases associated with the original index are restored with the remote index. If your application accesses the index using aliases, set this parameter to `true`. | Boolean | No | `false` +`ignore_index_settings` | A comma-separated list of index settings to ignore during the restore operation. For example, `index.refresh_interval,index.number_of_replicas`. This is useful when you want to apply different settings to the restored remote index than the ones configured in the original index. | String | No | Empty string +`number_of_replicas` | The number of replicas to configure for the restored remote index. This allows you to control replica allocation during the conversion process without requiring a separate update operation. Setting `number_of_replicas` during conversion helps prevent the cluster from entering a yellow state or creating unnecessary load during replica assignment. | Integer | No | `0` +`rename_pattern` | The naming pattern for the restored searchable snapshot index. Use `$1` as a placeholder for the original index name. For example, `remote_$1` renames `my-index` to `remote_my-index`. | String | No | `$1_remote` + +### Prerequisites + +Before using the `convert_index_to_remote` action, ensure the following: + +- A remote repository (S3, Azure, or GCS) is registered and accessible. +- A snapshot of the index exists in the specified repository, typically created using the `snapshot` action. +- The repository name matches the one used in the snapshot action. + +### Usage notes + +Note the following to ensure a smooth and predictable conversion when restoring an index as a searchable snapshot: + +- The original index is automatically deleted after the remote snapshot restore is successfully accepted. This ensures that only the searchable snapshot version remains, completing the conversion process. +- The repository name used in the `convert_index_to_remote` operation must match the repository name specified during the snapshot action. +- You can reference the snapshot using Mustache variables like `{% raw %}{{ctx.index}}{% endraw %}` or `{% raw %}{{ctx.indexUuid}}{% endraw %}` for dynamic naming. +- Consider your cluster's capacity when setting `number_of_replicas`. If there aren't enough eligible nodes for replica restoration, the cluster may enter a yellow state. + +### Basic example + +The following example shows a basic conversion using the minimum required parameters: + +```json +{ + "snapshot": { + "repository": "my_backup", + "snapshot": "{% raw %}{{ctx.index}}{% endraw %}" + }, + "convert_index_to_remote": { + "repository": "my_backup", + "snapshot": "{% raw %}{{ctx.index}}{% endraw %}" + } +} +``` +{% include copy.html %} + +### Advanced configuration example + +The following example demonstrates using all available configuration options. This configuration includes aliases, ignores certain index settings during restore, and configures two replicas for the searchable snapshot: + +```json +{ + "convert_index_to_remote": { + "repository": "my_backup", + "snapshot": "daily-snapshot", + "include_aliases": true, + "ignore_index_settings": "index.refresh_interval,index.number_of_replicas", + "number_of_replicas": 0, + "rename_pattern": "remote_$1" + } +} +``` +{% include copy.html %} + +### Complete policy example + +The following policy moves indexes older than 30 days to searchable snapshots with optimized settings for cost efficiency: + +```json +{ + "policy": { + "description": "Convert old indexes to searchable snapshots", + "default_state": "active", + "states": [ + { + "name": "active", + "actions": [], + "transitions": [ + { + "state_name": "archive", + "conditions": { + "min_index_age": "30d" + } + } + ] + }, + { + "name": "archive", + "actions": [ + { + "snapshot": { + "repository": "remote-repo", + "snapshot": "{% raw %}{{ctx.index}}{% endraw %}" + } + }, + { + "convert_index_to_remote": { + "repository": "remote-repo", + "snapshot": "{% raw %}{{ctx.index}}{% endraw %}", + "include_aliases": true, + "ignore_index_settings": "index.refresh_interval,index.number_of_replicas", + "number_of_replicas": 0 + } + } + ], + "transitions": [] + } + ] + } +} +``` +{% include copy.html %} + +## Index priority + +Set the priority for the index in a specific state. Unallocated shards of indexes are recovered in the order of their priority, whenever possible. The indexes with higher priority values are recovered first followed by the indexes with lower priority values. + +The `index_priority` operation has the following parameter. + +Parameter | Description | Type | Required | Default +:--- | :--- |:--- |:--- |:--- +`priority` | The priority for the index as soon as it enters a state. | Integer | Yes | 1 + +```json +"actions": [ + { + "index_priority": { + "priority": 50 + } + } +] +``` + +## Allocation + +Allocate the index to a node with a specific attribute set [like this]({{site.url}}{{site.baseurl}}/opensearch/cluster/#advanced-step-7-set-up-a-hot-warm-architecture). +For example, setting `require` to `warm` moves your data only to "warm" nodes. + +The `allocation` operation has the following parameters. At least one of `require`, `include`, or `exclude` must be specified. + +Parameter | Description | Type | Required +:--- | :--- |:--- |:--- +`require` | Allocate the index to a node with a specified attribute. | Object | No +`include` | Allocate the index to a node with any of the specified attributes. | Object | No +`exclude` | Don't allocate the index to a node with any of the specified attributes. | Object | No +`wait_for` | Wait for the policy to execute before allocating the index to a node with a specified attribute. | Boolean | No. Default is `false`. + +```json +"actions": [ + { + "allocation": { + "require": { "temp": "warm" } + } + } +] +``` + +## Rollup + +[Index rollup]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/index/) lets you periodically reduce data granularity by rolling up old data into summarized indexes. + +Rollup jobs can be continuous or non-continuous. A rollup job created using an ISM policy can only be non-continuous. +{: .note } + +## Stop replication + +Stops replication and converts the follower index to a regular index. + +```json +{ + "stop_replication": {} +} +``` + +When cross-cluster replication is enabled, the follower index becomes read-only, preventing all write operations. To manage replicated indexes on a follower cluster, you can perform the `stop_replication` action before performing other write operations. For example, you can define a policy that first runs `stop_replication` and then deletes the index by running a `delete` action. + +If security is enabled, in addition to [stop replication permissions]({{site.url}}{{site.baseurl}}/tuning-your-cluster/replication-plugin/permissions/#replication-permissions), you must have the `indices:internal/plugins/replication/index/stop` permission in order to use the `stop_replication` action. +{: .note} + +## Search only + +When an index enters `search_only` mode, OpenSearch removes its primary and regular replica shards while retaining search replicas for query operations. All write operations to the index are blocked. This is useful for log lifecycle management where older indexes no longer need write capability but should remain searchable. + +> This action requires the following prerequisites: +> - Remote store must be enabled on the cluster. +> - Segment replication must be enabled on the index. +> - Search replicas must be configured on the index. +> +> For more information about search-only mode and reader/writer separation, see [Separate index and search workloads]({{site.url}}{{site.baseurl}}/tuning-your-cluster/separate-index-and-search-workloads/). +{: .note} + +Set an index to search-only mode using the following action: + +```json +{ + "search_only": {} +} +``` + +If the index is already in search-only mode, the action completes successfully without making any changes. + +You can manually enable or disable `search_only` mode outside of ISM policies by calling the [Scale API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/scale/). +{: .tip} + +The following example policy transitions an index to `search_only` mode after 7 days: + +```json +{ + "policy": { + "policy_id": "hot-warm-search-only", + "default_state": "hot", + "states": [ + { + "name": "hot", + "actions": [], + "transitions": [ + { + "state_name": "warm", + "conditions": { + "min_index_age": "7d" + } + } + ] + }, + { + "name": "warm", + "actions": [ + { + "search_only": {} + } + ], + "transitions": [] + } + ] + } +} +``` +{% include copy-curl.html %} + +### Endpoints + +````bash +PUT _plugins/_rollup/jobs/{rollup_id} +GET _plugins/_rollup/jobs/{rollup_id} +DELETE _plugins/_rollup/jobs/{rollup_id} +POST _plugins/_rollup/jobs/{rollup_id}/_start +POST _plugins/_rollup/jobs/{rollup_id}/_stop +GET _plugins/_rollup/jobs/{rollup_id}/_explain +```` + +### Sample ISM rollup policy + +````json +{ + "policy": { + "description": "Sample rollup" , + "default_state": "rollup", + "states": [ + { + "name": "rollup", + "actions": [ + { + "rollup": { + "ism_rollup": { + "description": "Creating rollup through ISM", + "target_index": "target", + "target_index_settings":{ + "index.number_of_shards": 1, + "index.number_of_replicas": 1, + "index.codec": "best_compression" + }, + "page_size": 1000, + "dimensions": [ + { + "date_histogram": { + "fixed_interval": "60m", + "source_field": "order_date", + "target_field": "order_date", + "timezone": "America/Los_Angeles" + } + }, + { + "terms": { + "source_field": "customer_gender", + "target_field": "customer_gender" + } + }, + { + "terms": { + "source_field": "day_of_week", + "target_field": "day_of_week" + } + } + ], + "metrics": [ + { + "source_field": "taxless_total_price", + "metrics": [ + { + "sum": {} + } + ] + }, + { + "source_field": "total_quantity", + "metrics": [ + { + "avg": {} + }, + { + "max": {} + } + ] + } + ] + } + } + } + ], + "transitions": [] + } + ] + } +} +```` + +### Request body fields + +Request fields are required when creating an ISM policy. You can reference the [Index rollups API]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/api-rollup/#create-or-update-an-index-rollup-job) page for request field options. + +### Adding a rollup policy in Dashboards + +To add a rollup policy in Dashboards, follow the steps below. + +- Select the menu button on the upper-left of the Dashboards user interface. +- In the Dashboards menu, select `Index Management`. +- On the next screen select `Rollup jobs`. +- Select the `Create rollup` button. +- Follow the steps in the `Create rollup job` wizard. +- Add a name for the policy in the `Name` box. +- You can reference the [Index rollups API]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/api-rollup/#create-or-update-an-index-rollup-job) page to configure the rollup policy. +- Finally, select the `Create` button on the lower-right of the Dashboards user interface. diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md index 19c6952e684..5e86dc252df 100644 --- a/_im-plugin/ism/policies.md +++ b/_im-plugin/ism/policies.md @@ -1,51 +1,43 @@ --- layout: default title: Policies -nav_order: 1 +nav_order: 10 parent: Index State Management -has_children: false +has_children: true --- # ISM policies -Policies are JSON documents that define the following: +Policies are entities (stored as JSON documents) that define the following: - The *states* that an index can be in, including the default state for new indexes. For example, you might name your states "hot," "warm," "delete," and so on. For more information, see [States](#states). - Any *actions* that you want the plugin to take when an index enters a state, such as performing a rollover. For more information, see [Actions](#actions). - The conditions that must be met for an index to move into a new state, known as *transitions*. For example, if an index is more than eight weeks old, you might want to move it to the "delete" state. For more information, see [Transitions](#transitions). -In other words, a policy defines the *states* that an index can be in, the *actions* to perform when in a state, and the conditions that must be met to *transition* between states. +Actions and transitions are associated with states. A condition (such as index size or age) triggers a transition to a new state, and entering a state triggers its actions. You have complete flexibility in the way you can design your policies. You can create any state, transition to any other state, and specify any number of actions in each state. -This table lists the relevant fields of a policy. +The following table lists the fields of a policy. Field | Description | Type | Required | Read Only :--- | :--- |:--- |:--- | `policy_id` | The name of the policy. | String | Yes | Yes `description` | A human-readable description of the policy. | String | Yes | No -`ism_template` | Specify an ISM template to automatically apply the policy to the newly created index. | `nested list of objects` | No | No -`ism_template.index_patterns` | Specify a pattern that matches the newly created index name. | `list of strings` | No | No -`ism_template.priority` | Specify a priority to disambiguate when multiple policies match the newly created index name. | Integer | No | No +`ism_template` | An ISM template to automatically apply the policy to the newly created index. | `nested list of objects` | No | No +`ism_template.index_patterns` | A pattern that matches the newly created index name. | `list of strings` | No | No +`ism_template.priority` | A priority used to choose which policy to apply when multiple policies match a newly created index name. | Integer | No | No `last_updated_time` | The time the policy was last updated. | `timestamp` | Yes | Yes `error_notification` | The destination and message template for error notifications. The destination could be Amazon Chime, Slack, or a webhook URL. | `object` | No | No `default_state` | The default starting state for each index that uses this policy. | String | Yes | No `states` | The states that you define in the policy. | `nested list of objects` | Yes | No ---- - -#### Table of contents -1. TOC -{:toc} - - ---- ## States -A state is the description of the status that the managed index is currently in. A managed index can be in only one state at a time. Each state has associated actions that are executed sequentially on entering a state and transitions that are checked after all the actions have been completed. +A state defines the status of a managed index. A managed index can be in only one state at a time. A state's actions are executed sequentially on entering a state. A state's transitions are checked periodically after all the actions have been completed. -This table lists the parameters that you can define for a state. +The following table lists the parameters that you can define for a state. Field | Description | Type | Required :--- | :--- |:--- |:--- | @@ -53,17 +45,25 @@ Field | Description | Type | Required `actions` | The actions to execute after entering a state. For more information, see [Actions](#actions). | `nested list of objects` | Yes `transitions` | The next states and the conditions required to transition to those states. If no transitions exist, the policy assumes that it's complete and can now stop managing the index. For more information, see [Transitions](#transitions). | `nested list of objects` | Yes ---- ## Actions -Actions are the steps that the policy sequentially executes on entering a specific state. +Actions are [operations]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies-operations) that a policy can execute upon entering a specific state. -ISM executes actions in the order in which they are defined. For example, if you define actions `[A,B,C,D]`, ISM executes action `A`, and then goes into a sleep period based on the cluster setting `plugins.index_state_management.job_interval`. Once the sleep period ends, ISM continues to execute the remaining actions. However, if ISM cannot successfully execute action `A`, the operation ends, and actions `B`, `C`, and `D` do not get executed. +ISM executes actions in the order in which they are defined. If an action fails, the state actions are abandoned, and remaining actions are not executed. + +For example, if you define actions `[A,B,C,D]`, ISM does the following: +1. Executes action `A`. +2. Sleeps for a period based on the cluster setting `plugins.index_state_management.job_interval`. +3. Executes action `B`. + +And so on. + +If ISM cannot successfully execute action `A`, actions `B`, `C`, and `D` do not get executed. Optionally, you can define an action's timeout period, which, if exceeded, forcibly fails the action. For example, if timeout is set to `1d`, and ISM has not completed the action within one day, even after retries, the action fails. -This table lists the parameters that you can define for an action. +The following table lists the parameters that you can define for an action. Parameter | Description | Type | Required | Default :--- | :--- |:--- |:--- | @@ -78,696 +78,35 @@ Parameter | Description | Type | Required | Default `backoff` | The backoff policy type to use when retrying. Valid values are Exponential, Constant, and Linear. | String | No | Exponential `delay` | The time to wait between retries. Accepts time units for minutes, hours, and days. | `time unit` | No | 1 minute -The following example action has a timeout period of one hour. The policy retries this action three times with an exponential backoff policy, with a delay of 10 minutes between each retry: - -```json -"actions": { - "timeout": "1h", - "retry": { - "count": 3, - "backoff": "exponential", - "delay": "10m" - } -} -``` - -For a list of available unit types, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). - -## ISM supported operations - -ISM supports the following operations: - -- [Force merge](#force-merge) -- [Read only](#read-only) -- [Read write](#read-write) -- [Replica count](#replica-count) -- [Shrink](#shrink) -- [Close](#close) -- [Open](#open) -- [Delete](#delete) -- [Rollover](#rollover) -- [Notification](#notification) -- [Snapshot](#snapshot) -- [Convert index to remote](#convert-index-to-remote) -- [Index priority](#index-priority) -- [Allocation](#allocation) -- [Rollup](#rollup) -- [Stop replication](#stop-replication) -- [Search only](#search-only) - -### Force merge - -Reduces the number of Lucene segments by merging the segments of individual shards. This operation attempts to set the index to a `read-only` state before starting the merging process. - -Parameter | Description | Type | Required -:--- | :--- |:--- |:--- | -`max_num_segments` | The number of segments to reduce the shard to. | Integer | Yes - -```json -{ - "force_merge": { - "max_num_segments": 1 - } -} -``` - -### Read only - -Sets a managed index to be read only. - -```json -{ - "read_only": {} -} -``` - -Set the index setting `index.blocks.write` to `true` for a managed index. ***Note:** this block does not prevent the index from refreshing. - -### Read write - -Sets a managed index to be writeable. - -```json -{ - "read_write": {} -} -``` - -### Replica count - -Sets the number of replicas to assign to an index. - -Parameter | Description | Type | Required -:--- | :--- |:--- |:--- | -`number_of_replicas` | Defines the number of replicas to assign to an index. | Integer | Yes - -```json -{ - "replica_count": { - "number_of_replicas": 2 - } -} -``` - -For information about setting replicas, see [Primary and replica shards]({{site.url}}{{site.baseurl}}/intro/#primary-and-replica-shards). - -### Shrink - -Allows you to reduce the number of primary shards in your indexes. With this action, you can specify: - -- The number of primary shards that the target index should contain. -- A max shard size for the primary shards in the target index. -- Specify a percentage to shrink the number of primary shards in the target index. - -```json -"shrink": { - "num_new_shards": 1, - "target_index_name_template": { - "source": "{{ctx.index}}_shrunken" - }, - "aliases": [ - { - "my-alias": {} - } - ], - "switch_aliases": true, - "force_unsafe": false -} -``` - -Parameter | Description | Type | Example | Required -:--- | :--- |:--- |:--- | -`num_new_shards` | The maximum number of primary shards in the shrunken index. | Integer | `5` | Yes. It, however, cannot be used with `max_shard_size` or `percentage_of_source_shards`. -`max_shard_size` | The maximum size in bytes of a shard for the target index. | Keyword | `5gb` | Yes, however, it cannot be used with `num_new_shards` or `percentage_of_source_shards`. -`percentage_of_source_shards` | Percentage of the number of original primary shards to shrink. This parameter indicates the minimum percentage to use when shrinking the number of primary shards. Must be between 0.0 and 1.0, exclusive. | Percentage | `0.5` | Yes, however it cannot be used with `max_shard_size` or `num_new_shards` -`target_index_name_template` | The name of the shrunken index. Accepts strings and the Mustache variables `{% raw %}{{ctx.index}}{% endraw %}` and `{% raw %}{{ctx.indexUuid}}{% endraw %}`. | String or Mustache template | `{"source": "{% raw %}{{ctx.index}}_shrunken"}{% endraw %}` | No -`aliases` | Aliases to add to the new index. | Object | `myalias` | No. It must be an array of alias objects. -`switch_aliases` | If `true`, copies the aliases from the source index to the target index. If there is a name conflict with an alias from the `aliases` field, the alias in the `aliases` field is used instead of the name. | Boolean | `true` | No. The default implicit value is `false`, which means no aliases are copied by default. -`force_unsafe` | If `true`, shrinks the index even if it has no replicas. | Boolean | `false` | No - -If you want to add `aliases` to the action, the parameter must include an array of [alias objects]({{site.url}}{{site.baseurl}}/api-reference/alias/). For example, - -```json -"aliases": [ - { - "my-alias": {} - }, - { - "my-second-alias": { - "is_write_index": false, - "filter": { - "multi_match": { - "query": "QUEEN", - "fields": ["speaker", "text_entry"] - } - }, - "index_routing" : "1", - "search_routing" : "1" - } - }, -] -``` - -### Close - -Closes the managed index. - -```json -{ - "close": {} -} -``` - -Closed indexes remain on disk, but consume no CPU or memory. You can't read from, write to, or search closed indexes. - -Closing an index is a good option if you need to retain data for longer than you need to actively search it and have sufficient disk space on your data nodes. If you need to search the data again, reopening a closed index is simpler than restoring an index from a snapshot. - -### Open - -Opens a managed index. - -```json -{ - "open": {} -} -``` - -### Delete - -Deletes a managed index. - -```json -{ - "delete": {} -} -``` - -### Rollover - -Rolls an alias over to a new index when the managed index meets one of the rollover conditions. - -

- -> **IMPORTANT** -> ->ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will be performed if the value **has reached** or _has exceeded_ the configured limit **when the check is performed**. For example, with `min_size` configured to a value of 100 GiB, ISM might check the index at 99 GiB and not perform the rollover. However, if the index has grown past the limit by the next check (for example, to 105 GiB), the operation is performed. -{: .important} - -If you need to skip the rollover action, you can set the index setting `index.plugins.index_state_management.rollover_skip` to `true`. For example, if you receive the error message "Missing alias or not the write index...", you can set the `index.plugins.index_state_management.rollover_skip` parameter to `true` and retry to skip rollover action. - -The index format must match the pattern: `^.*-\d+$`. For example, `(logs-000001)`. -Set `index.plugins.index_state_management.rollover_alias` as the alias to rollover. - -Parameter | Description | Type | Example | Required -:--- | :--- |:--- |:--- | -`min_size` | The minimum size of the total primary shard storage (not counting replicas) required to roll over the index. For example, if you set `min_size` to 100 GiB and your index has 5 primary shards and 5 replica shards of 20 GiB each, the total size of all primary shards is 100 GiB, so the rollover occurs. See [**Important** note](#important-note). | String | `20gb` or `5mb` | No -`min_primary_shard_size` | The minimum storage size of a **single primary shard** required to roll over the index. For example, if you set `min_primary_shard_size` to 30 GiB and **one of** the primary shards in the index has a size greater than the condition, the rollover occurs. See [**Important** note](#important-note). | String | `20gb` or `5mb` | No -`min_doc_count` | The minimum number of documents required to roll over the index. See [**Important** note](#important-note). | Integer | `2000000` | No -`min_index_age` | The minimum age required to roll over the index. Index age is the time between its creation and the present. Supported units are `d` (days), `h` (hours), `m` (minutes), `s` (seconds), `ms` (milliseconds), and `micros` (microseconds). See [**Important** note](#important-note). | String | `5d` or `7h` | No -`copy_alias` | Controls whether to copy over all aliases from the current index to a newly created index. Defaults to `false`. | `boolean` | `true` or `false` | No - -```json -{ - "rollover": { - "min_size": "50gb" - } -} -``` - -```json -{ - "rollover": { - "min_primary_shard_size": "30gb" - } -} -``` - -```json -{ - "rollover": { - "min_doc_count": 100000000 - } -} -``` - -```json -{ - "rollover": { - "min_index_age": "30d" - } -} -``` - -### Notification - -Sends you a notification. - -Parameter | Description | Type | Required -:--- | :--- |:--- |:--- | -`destination` | The destination URL. | `Slack, Amazon Chime, or webhook URL` | Yes -`message_template` | The text of the message. You can add variables to your messages using [Mustache templates](https://mustache.github.io/mustache.5.html). | `object` | Yes - -The destination system **must** return a response otherwise the notification operation throws an error. - -#### Example 1: Chime notification - -```json -{ - "notification": { - "destination": { - "chime": { - "url": "" - } - }, - "message_template": { - "source": "the index is {% raw %}{{ctx.index}}{% endraw %}" - } - } -} -``` - -#### Example 2: Custom webhook notification - -```json -{ - "notification": { - "destination": { - "custom_webhook": { - "url": "https://" - } - }, - "message_template": { - "source": "the index is {% raw %}{{ctx.index}}{% endraw %}" - } - } -} -``` - -#### Example 3: Slack notification - -```json -{ - "notification": { - "destination": { - "slack": { - "url": "https://hooks.slack.com/services/xxx/xxxxxx" - } - }, - "message_template": { - "source": "the index is {% raw %}{{ctx.index}}{% endraw %}" - } - } -} -``` - -You can use `ctx` variables in your message to represent a number of policy parameters based on the past executions of your policy. For example, if your policy has a rollover action, you can use `{% raw %}{{ctx.action.name}}{% endraw %}` in your message to represent the name of the rollover. - -The following `ctx` variable options are available for every policy: - -#### Guaranteed variables - -Parameter | Description | Type -:--- | :--- |:--- |:--- | -`index` | The name of the index. | String -`index_uuid` | The UUID of the index. | String -`policy_id` | The name of the policy. | String - -### Snapshot - -Back up your cluster’s indexes and state. For more information about snapshots, see [Take and restore snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore/). - -The `snapshot` operation has the following parameters. - -Parameter | Description | Type | Required | Default -:--- | :--- |:--- |:--- | -`repository` | The repository name that you register through the native snapshot API operations. | String | Yes | - -`snapshot` | The name of the snapshot. Accepts strings and the Mustache variables `{% raw %}{{ctx.index}}{% endraw %}` and `{% raw %}{{ctx.indexUuid}}{% endraw %}`. If the Mustache variables are invalid, then the snapshot name defaults to the index's name. | String or Mustache template | Yes | - - -```json -{ - "snapshot": { - "repository": "my_backup", - "snapshot": "{% raw %}{{ctx.indexUuid}}{% endraw %}" - } -} -``` - -### Convert index to remote - -Converts an existing index into a searchable snapshot by restoring it from a remote snapshot repository. This action reduces storage costs by moving infrequently accessed data to remote storage while keeping it searchable. After the restore request is accepted, the original index is automatically deleted, ensuring that only the remote snapshot-backed index remains. - -The `convert_index_to_remote` operation has the following parameters. - -Parameter | Description | Type | Required | Default -:--- | :--- |:--- |:--- | -`repository` | The repository name registered through the native snapshot API operations. Must be a remote repository (for example, S3, Azure, or GCS). | String | Yes | N/A -`snapshot` | The name of the snapshot created by the snapshot action. | String | Yes | N/A -`include_aliases` | Whether to include index aliases during the restore operation. If `true`, all aliases associated with the original index are restored with the remote index. If your application accesses the index using aliases, set this parameter to `true`. | Boolean | No | `false` -`ignore_index_settings` | A comma-separated list of index settings to ignore during the restore operation. For example, `index.refresh_interval,index.number_of_replicas`. This is useful when you want to apply different settings to the restored remote index than the ones configured in the original index. | String | No | Empty string -`number_of_replicas` | The number of replicas to configure for the restored remote index. This allows you to control replica allocation during the conversion process without requiring a separate update operation. Setting `number_of_replicas` during conversion helps prevent the cluster from entering a yellow state or creating unnecessary load during replica assignment. | Integer | No | `0` -`rename_pattern` | The naming pattern for the restored searchable snapshot index. Use `$1` as a placeholder for the original index name. For example, `remote_$1` renames `my-index` to `remote_my-index`. | String | No | `$1_remote` - -#### Prerequisites - -Before using the `convert_index_to_remote` action, ensure the following: - -- A remote repository (S3, Azure, or GCS) is registered and accessible. -- A snapshot of the index exists in the specified repository, typically created using the `snapshot` action. -- The repository name matches the one used in the snapshot action. - -#### Usage notes - -Note the following to ensure a smooth and predictable conversion when restoring an index as a searchable snapshot: - -- The original index is automatically deleted after the remote snapshot restore is successfully accepted. This ensures that only the searchable snapshot version remains, completing the conversion process. -- The repository name used in the `convert_index_to_remote` operation must match the repository name specified during the snapshot action. -- You can reference the snapshot using Mustache variables like `{% raw %}{{ctx.index}}{% endraw %}` or `{% raw %}{{ctx.indexUuid}}{% endraw %}` for dynamic naming. -- Consider your cluster's capacity when setting `number_of_replicas`. If there aren't enough eligible nodes for replica restoration, the cluster may enter a yellow state. - -#### Basic example - -The following example shows a basic conversion using the minimum required parameters: - -```json -{ - "snapshot": { - "repository": "my_backup", - "snapshot": "{% raw %}{{ctx.index}}{% endraw %}" - }, - "convert_index_to_remote": { - "repository": "my_backup", - "snapshot": "{% raw %}{{ctx.index}}{% endraw %}" - } -} -``` -{% include copy.html %} - -#### Advanced configuration example - -The following example demonstrates using all available configuration options. This configuration includes aliases, ignores certain index settings during restore, and configures two replicas for the searchable snapshot: - -```json -{ - "convert_index_to_remote": { - "repository": "my_backup", - "snapshot": "daily-snapshot", - "include_aliases": true, - "ignore_index_settings": "index.refresh_interval,index.number_of_replicas", - "number_of_replicas": 0, - "rename_pattern": "remote_$1" - } -} -``` -{% include copy.html %} - -#### Complete policy example - -The following policy moves indexes older than 30 days to searchable snapshots with optimized settings for cost efficiency: +### Example action -```json -{ - "policy": { - "description": "Convert old indexes to searchable snapshots", - "default_state": "active", - "states": [ - { - "name": "active", - "actions": [], - "transitions": [ - { - "state_name": "archive", - "conditions": { - "min_index_age": "30d" - } - } - ] - }, - { - "name": "archive", - "actions": [ - { - "snapshot": { - "repository": "remote-repo", - "snapshot": "{% raw %}{{ctx.index}}{% endraw %}" - } - }, - { - "convert_index_to_remote": { - "repository": "remote-repo", - "snapshot": "{% raw %}{{ctx.index}}{% endraw %}", - "include_aliases": true, - "ignore_index_settings": "index.refresh_interval,index.number_of_replicas", - "number_of_replicas": 0 - } - } - ], - "transitions": [] - } - ] - } -} -``` -{% include copy.html %} - -### Index priority - -Set the priority for the index in a specific state. Unallocated shards of indexes are recovered in the order of their priority, whenever possible. The indexes with higher priority values are recovered first followed by the indexes with lower priority values. - -The `index_priority` operation has the following parameter. - -Parameter | Description | Type | Required | Default -:--- | :--- |:--- |:--- |:--- -`priority` | The priority for the index as soon as it enters a state. | Integer | Yes | 1 +The following example `read_only` action has a timeout period of one hour. The policy retries this action three times with an exponential backoff policy, with a delay of 10 minutes between each retry: ```json "actions": [ { - "index_priority": { - "priority": 50 - } - } -] -``` - -### Allocation - -Allocate the index to a node with a specific attribute set [like this]({{site.url}}{{site.baseurl}}/opensearch/cluster/#advanced-step-7-set-up-a-hot-warm-architecture). -For example, setting `require` to `warm` moves your data only to "warm" nodes. - -The `allocation` operation has the following parameters. At least one of `require`, `include`, or `exclude` must be specified. - -Parameter | Description | Type | Required -:--- | :--- |:--- |:--- -`require` | Allocate the index to a node with a specified attribute. | Object | No -`include` | Allocate the index to a node with any of the specified attributes. | Object | No -`exclude` | Don't allocate the index to a node with any of the specified attributes. | Object | No -`wait_for` | Wait for the policy to execute before allocating the index to a node with a specified attribute. | Boolean | No. Default is `false`. - -```json -"actions": [ - { - "allocation": { - "require": { "temp": "warm" } - } + "timeout": "1h", + "retry": { + "count": 3, + "backoff": "exponential", + "delay": "10m" + }, + "read_only": {} } ] ``` -### Rollup - -[Index rollup]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/index/) lets you periodically reduce data granularity by rolling up old data into summarized indexes. - -Rollup jobs can be continuous or non-continuous. A rollup job created using an ISM policy can only be non-continuous. -{: .note } - -### Stop replication - -Stops replication and converts the follower index to a regular index. - -```json -{ - "stop_replication": {} -} -``` - -When cross-cluster replication is enabled, the follower index becomes read-only, preventing all write operations. To manage replicated indexes on a follower cluster, you can perform the `stop_replication` action before performing other write operations. For example, you can define a policy that first runs `stop_replication` and then deletes the index by running a `delete` action. - -If security is enabled, in addition to [stop replication permissions]({{site.url}}{{site.baseurl}}/tuning-your-cluster/replication-plugin/permissions/#replication-permissions), you must have the `indices:internal/plugins/replication/index/stop` permission in order to use the `stop_replication` action. -{: .note} - -### Search only - -When an index enters `search_only` mode, OpenSearch removes its primary and regular replica shards while retaining search replicas for query operations. All write operations to the index are blocked. This is useful for log lifecycle management where older indexes no longer need write capability but should remain searchable. - -> This action requires the following prerequisites: -> - Remote store must be enabled on the cluster. -> - Segment replication must be enabled on the index. -> - Search replicas must be configured on the index. -> -> For more information about search-only mode and reader/writer separation, see [Separate index and search workloads]({{site.url}}{{site.baseurl}}/tuning-your-cluster/separate-index-and-search-workloads/). -{: .note} - -Set an index to search-only mode using the following action: - -```json -{ - "search_only": {} -} -``` - -If the index is already in search-only mode, the action completes successfully without making any changes. - -You can manually enable or disable `search_only` mode outside of ISM policies by calling the [Scale API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/scale/). -{: .tip} - -The following example policy transitions an index to `search_only` mode after 7 days: - -```json -{ - "policy": { - "policy_id": "hot-warm-search-only", - "default_state": "hot", - "states": [ - { - "name": "hot", - "actions": [], - "transitions": [ - { - "state_name": "warm", - "conditions": { - "min_index_age": "7d" - } - } - ] - }, - { - "name": "warm", - "actions": [ - { - "search_only": {} - } - ], - "transitions": [] - } - ] - } -} -``` -{% include copy-curl.html %} - -#### Endpoints - -````bash -PUT _plugins/_rollup/jobs/{rollup_id} -GET _plugins/_rollup/jobs/{rollup_id} -DELETE _plugins/_rollup/jobs/{rollup_id} -POST _plugins/_rollup/jobs/{rollup_id}/_start -POST _plugins/_rollup/jobs/{rollup_id}/_stop -GET _plugins/_rollup/jobs/{rollup_id}/_explain -```` - -#### Sample ISM rollup policy - -````json -{ - "policy": { - "description": "Sample rollup" , - "default_state": "rollup", - "states": [ - { - "name": "rollup", - "actions": [ - { - "rollup": { - "ism_rollup": { - "description": "Creating rollup through ISM", - "target_index": "target", - "target_index_settings":{ - "index.number_of_shards": 1, - "index.number_of_replicas": 1, - "index.codec": "best_compression" - }, - "page_size": 1000, - "dimensions": [ - { - "date_histogram": { - "fixed_interval": "60m", - "source_field": "order_date", - "target_field": "order_date", - "timezone": "America/Los_Angeles" - } - }, - { - "terms": { - "source_field": "customer_gender", - "target_field": "customer_gender" - } - }, - { - "terms": { - "source_field": "day_of_week", - "target_field": "day_of_week" - } - } - ], - "metrics": [ - { - "source_field": "taxless_total_price", - "metrics": [ - { - "sum": {} - } - ] - }, - { - "source_field": "total_quantity", - "metrics": [ - { - "avg": {} - }, - { - "max": {} - } - ] - } - ] - } - } - } - ], - "transitions": [] - } - ] - } -} -```` - -#### Request body fields - -Request fields are required when creating an ISM policy. You can reference the [Index rollups API]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/rollup-api/#create-or-update-an-index-rollup-job) page for request field options. - -#### Adding a rollup policy in Dashboards - -To add a rollup policy in Dashboards, follow the following steps. - -- Select the menu button on the upper-left of the Dashboards user interface. -- In the Dashboards menu, select `Index Management`. -- On the next screen select `Rollup jobs`. -- Select the `Create rollup` button. -- Follow the steps in the `Create rollup job` wizard. -- Add a name for the policy in the `Name` box. -- You can reference the [Index rollups API]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/rollup-api/#create-or-update-an-index-rollup-job) page to configure the rollup policy. -- Finally, select the `Create` button on the lower-right of the Dashboards user interface. - ---- +For a list of available unit types, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). ## Transitions -Transitions define the conditions that need to be met for a state to change. After all actions in the current state are completed, the policy starts checking the conditions for transitions. +Transitions define the conditions that trigger a state to change. After all actions in the current state are completed, the policy starts checking the conditions for transitions. + +ISM evaluates transitions in the order in which they are defined. It uses the first transition that evaluates to `true`. -ISM evaluates transitions in the order in which they are defined. For example, if you define transitions: `[A,B,C,D]`, ISM iterates through this list of transitions until it finds a transition that evaluates to `true`, it then stops and sets the next state to the one defined in that transition. On its next execution, ISM dismisses the rest of the transitions and starts in that new state. +If you don't specify any conditions in a transition, then it always evaluates to `true`. If the policy checks such a transition, it immediately transitions the index to the state defined in the transition. -If you don't specify any conditions in a transition and leave it empty, then it's assumed to be the equivalent of always true. This means that the policy transitions the index to this state the moment it checks. +For example, assume you've defined the transitions: `[A,B,C,D]`, and that transitions `A`, `B`, and `C` currently evaluate to `false` and that `D` has no conditions. ISM iterates through the list in order and sets the next state to the one defined in transition `D`. On its next execution, ISM starts in the state defined by `D`. This table lists the parameters you can define for transitions. @@ -788,7 +127,7 @@ Parameter | Description | Type | Required `no_alias` | Controls transition based on alias presence. If `true`, transition occurs only when the index has **no aliases**. If `false`, transition occurs only when at least **one alias exists**. | `boolean` | No `cron` | The `cron` job that triggers the transition if no other transition happens first. | `object` | No `cron.cron.expression` | The `cron` expression that triggers the transition. | String | Yes -`cron.cron.timezone` | The time zone that triggers the transition. | String | Yes +`cron.cron.timezone` | The time zone for the triggering `cron` expression triggers the transition. | String | Yes All time-based values (`min_index_age`, `min_rollover_age`, `min_state_age`) use [standard OpenSearch time units]({{site.url}}{{site.baseurl}}/api-reference/common-parameters/#time-units). {: .note} @@ -807,7 +146,7 @@ The following example transitions the index to a `cold` state after a period of ] ``` -ISM checks the conditions on every execution of the policy based on the set interval. +ISM checks the conditions on every execution of the policy based on the `job_interval` [setting]({{site.url}}{{site.baseurl}}/im-plugin/ism/settings/). This example uses the `cron` condition to transition indexes every Saturday at 5:00 PT: @@ -827,20 +166,16 @@ This example uses the `cron` condition to transition indexes every Saturday at 5 ] ``` -Note that this condition does not execute at exactly 5:00 PM; the job still executes based off the `job_interval` setting. Due to this variance in start time and the amount of time that it can take for actions to complete prior to checking transition conditions, we recommend against overly narrow cron expressions. For example, don't use `15 17 * * SAT` (5:15 PM on Saturday). +Note that this condition does not execute at exactly 5:00 PM; the job still executes as defined by the `job_interval` setting. Due to this variance in start time and the amount of time that it can take for actions to complete prior to checking transition conditions, we recommend against overly narrow cron expressions. For example, don't use `15 17 * * SAT` (5:15 PM on Saturday). -A window of an hour, which this example uses, is generally sufficient, but you might increase it to 2--3 hours to avoid missing the window and having to wait a week for the transition to occur. Alternately, you could use a broader expression such as `* * * * SAT,SUN` to have the transition occur at any time during the weekend. +A window of an hour, which this example uses, is generally sufficient, but you might increase it to 2 or 3 hours to avoid missing the window and having to wait a week for the transition to occur. Alternately, you could use a broader expression such as `* * * * SAT,SUN` to have the transition occur at any time during the weekend. -For information about writing cron expressions, see [Cron expression reference]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/). +For information about writing cron expressions, see the [Cron expression reference]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/). ---- ## Error notifications -The `error_notification` operation sends you a notification if your managed index fails. -It notifies a single destination or [notification channel]({{site.url}}{{site.baseurl}}/notifications-plugin/index/) with a custom message. - -Set up error notifications at the policy level: +An error notification can be set up at the policy level as shown in the following example: ```json { @@ -854,6 +189,10 @@ Set up error notifications at the policy level: } ``` +An `error_notification` sends a notification if your managed index fails. + +An error notification sends to a single destination or [notification channel]({{site.url}}{{site.baseurl}}/notifications-plugin/index/) with a custom message. + Parameter | Description | Type | Required :--- | :--- |:--- |:--- | `destination` | The destination URL. | `Slack, Amazon Chime, or webhook URL` | Yes if `channel` isn't specified @@ -862,7 +201,7 @@ Parameter | Description | Type | Required The destination system **must** return a response otherwise the `error_notification` operation throws an error. -#### Example 1: Chime notification +### Example 1: Chime notification ```json { @@ -879,7 +218,7 @@ The destination system **must** return a response otherwise the `error_notificat } ``` -#### Example 2: Custom webhook notification +### Example 2: Custom webhook notification ```json { @@ -896,7 +235,7 @@ The destination system **must** return a response otherwise the `error_notificat } ``` -#### Example 3: Slack notification +### Example 3: Slack notification ```json { @@ -913,7 +252,7 @@ The destination system **must** return a response otherwise the `error_notificat } ``` -#### Example 4: Using a notification channel +### Example 4: Using a notification channel ```json { @@ -928,300 +267,5 @@ The destination system **must** return a response otherwise the `error_notificat } ``` -You can use the same options for `ctx` variables as the [Notification](#notification) operation. - -## Sample policy with ISM template for auto rollover - -The following sample template policy is for a rollover use case. - -If you want to skip rollovers for an index, set `index.plugins.index_state_management.rollover_skip` to `true` in the settings of that index. - -1. Create a policy with an `ism_template` field: - - ```json - PUT _plugins/_ism/policies/rollover_policy - { - "policy": { - "description": "Example rollover policy.", - "default_state": "rollover", - "states": [ - { - "name": "rollover", - "actions": [ - { - "rollover": { - "min_doc_count": 1 - } - } - ], - "transitions": [] - } - ], - "ism_template": { - "index_patterns": ["log*"], - "priority": 100 - } - } - } - ``` - {% include copy-curl.html %} - - You need to specify the `index_patterns` field. If you don't specify a value for `priority`, it defaults to 0. - -2. Set up a template with the `rollover_alias` as `log` : - - ```json - PUT _index_template/ism_rollover - { - "index_patterns": ["log*"], - "template": { - "settings": { - "plugins.index_state_management.rollover_alias": "log" - } - } - } - ``` - {% include copy-curl.html %} - -3. Create an index with the `log` alias: - - ```json - PUT log-000001 - { - "aliases": { - "log": { - "is_write_index": true - } - } - } - ``` - {% include copy-curl.html %} - -4. Index a document to trigger the rollover condition: - - ```json - POST log/_doc - { - "message": "dummy" - } - ``` - {% include copy-curl.html %} - -5. Verify if the policy is attached to the `log-000001` index: - - ```json - GET _plugins/_ism/explain/log-000001?pretty - ``` - {% include copy-curl.html %} - -## Example policy with ISM templates for the alias action - -The following example policy is for an alias action use case. - -In the following example, the first job will trigger the rollover action, and a new index will be created. Next, another document is added to the two indexes. The new job will then cause the second index to point to the log alias, and the older index will be removed due to the alias action. - -First, create an ISM policy: - -```json -PUT /_plugins/_ism/policies/rollover_policy?pretty -{ - "policy": { - "description": "Example rollover policy.", - "default_state": "rollover", - "states": [ - { - "name": "rollover", - "actions": [ - { - "rollover": { - "min_doc_count": 1 - } - } - ], - "transitions": [{ - "state_name": "alias", - "conditions": { - "min_doc_count": "2" - } - }] - }, - { - "name": "alias", - "actions": [ - { - "alias": { - "actions": [ - { - "remove": { - "alias": "log" - } - } - ] - } - } - ] - } - ], - "ism_template": { - "index_patterns": ["log*"], - "priority": 100 - } - } -} -``` -{% include copy-curl.html %} - -Next, create an index template on which to enable the policy: - -```json -PUT /_index_template/ism_rollover? -{ - "index_patterns": ["log*"], - "template": { - "settings": { - "plugins.index_state_management.rollover_alias": "log" - } - } -} -``` -{% include copy-curl.html %} - -Next, change the cluster settings to trigger jobs every minute: - -```json -PUT /_cluster/settings?pretty=true -{ - "persistent" : { - "plugins.index_state_management.job_interval" : 1 - } -} -``` -{% include copy-curl.html %} - -Next, create a new index: - -```json -PUT /log-000001 -{ - "aliases": { - "log": { - "is_write_index": true - } - } -} -``` -{% include copy-curl.html %} - -Finally, add a document to the index to trigger the job: - -```json -POST /log-000001/_doc -{ - "message": "dummy" -} -``` -{% include copy-curl.html %} - -You can verify these steps using the Alias and Index API: - -```json -GET /_cat/indices?pretty -``` -{% include copy-curl.html %} - -```json -GET /_cat/aliases?pretty -``` -{% include copy-curl.html %} - -Note: The `index` and `remove_index` parameters are not allowed with alias action policies. Only the `add` and `remove` alias action parameters are allowed. -{: .warning } - -## Example policy - -The following example policy implements a `hot`, `warm`, and `delete` workflow. You can use this policy as a template to prioritize resources to your indexes based on their levels of activity. - -In this case, an index is initially in a `hot` state. After 7 days, it changes to a `warm` state, where the number of replicas is reduced to 1 and the indexes are moved to nodes with the `warm` attribute. - -After 30 days, the policy moves this index into a `delete` state. The service sends a notification to a Chime room that the index is being deleted, and then permanently deletes it. - -```json -{ - "policy": { - "description": "hot warm delete workflow", - "default_state": "hot", - "schema_version": 1, - "states": [ - { - "name": "hot", - "actions": [ - { - "rollover": { - "min_index_age": "7d", - "min_primary_shard_size": "30gb" - } - } - ], - "transitions": [ - { - "state_name": "warm" - } - ] - }, - { - "name": "warm", - "actions": [ - { - "replica_count": { - "number_of_replicas": 1 - } - }, - { - "allocation": { - "require": { - "temp": "warm" - } - } - } - ], - "transitions": [ - { - "state_name": "delete", - "conditions": { - "min_index_age": "30d" - } - } - ] - }, - { - "name": "delete", - "actions": [ - { - "notification": { - "destination": { - "chime": { - "url": "" - } - }, - "message_template": { - "source": "The index {% raw %}{{ctx.index}}{% endraw %} is being deleted" - } - } - }, - { - "delete": {} - } - ] - } - ], - "ism_template": { - "index_patterns": ["log*"], - "priority": 100 - } - } -} -``` - -This diagram shows the `states`, `transitions`, and `actions` of the preceding policy as a finite-state machine. For more information about finite-state machines, see [Wikipedia](https://en.wikipedia.org/wiki/Finite-state_machine). +You can use the same options for `ctx` variables as the [Notification]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies-operations/#notification) operation. -![Policy State Machine]({{site.url}}{{site.baseurl}}/images/ism.png) diff --git a/_im-plugin/ism/settings.md b/_im-plugin/ism/settings.md index 2f876295696..ab2a6995200 100644 --- a/_im-plugin/ism/settings.md +++ b/_im-plugin/ism/settings.md @@ -2,14 +2,14 @@ layout: default title: Settings parent: Index State Management -nav_order: 4 +nav_order: 50 --- # ISM settings We don't recommend changing these settings; the defaults should work well for most use cases. -Index State Management (ISM) stores its configuration in the `.opendistro-ism-config` index. Don't modify this index without using the [ISM API operations]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/). +Index State Management (ISM) stores its configuration in the `.opendistro-ism-config` index. Don't modify this index without using the [ISM API operations]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/). All settings are available using the OpenSearch `_cluster/settings` operation. None require a restart, and all can be marked `persistent` or `transient`. To learn more about static and dynamic settings, see [Configuring OpenSearch]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/). diff --git a/_im-plugin/notifications-settings.md b/_im-plugin/notifications/api-notifications.md similarity index 88% rename from _im-plugin/notifications-settings.md rename to _im-plugin/notifications/api-notifications.md index aeccb3c196e..9fdd819256c 100644 --- a/_im-plugin/notifications-settings.md +++ b/_im-plugin/notifications/api-notifications.md @@ -1,17 +1,16 @@ --- layout: default -title: Notification settings -nav_order: 100 +title: With the API +parent: Index notifications +nav_order: 20 --- -# Notification settings +# Setting up notifications the API Introduced 2.8 {: .label .label-purple } -You can use notification settings to configure notifications about long-running index operations. Set up automatic [notifications]({{site.url}}{{site.baseurl}}/observing-your-data/notifications/index/) when long-running index operations are complete by [using Notifications in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/notifications/) or through the API. - -Configuring notification settings is useful for long-running index operations, such as `open`, `reindex`, `resize`, and `force merge`. When you send a request for those operations and set the `wait_for_completion` parameter to `false`, the operation returns immediately and the response contains a task ID. You can use that task ID to configure notifications for this operation. +Use the Notifications API to configure notifications about long-running index operations. When you send a request for those operations and set the `wait_for_completion` parameter to `false`, the operation returns immediately and the response contains a task ID. You can use that task ID to configure notifications for this operation. ## Configuring notification settings @@ -32,7 +31,7 @@ The following table lists the parameters for long-running index operation notifi | `lron_condition.failure` | Boolean | Set this parameter to `true` to be notified when the operation fails or times out. Optional. Default is `true`. | | `channels` | Object | Supported communication channels include Amazon Chime, Amazon Simple Notification Service (Amazon SNS), Amazon Simple Email Service (Amazon SES), email through SMTP, Slack, and custom webhooks. If either `lron_condition.success` or `lron_condition.failure` is `true`, `channels` must contain at least one channel. Learn how to configure notification channels in [Notifications]({{site.url}}{{site.baseurl}}/observing-your-data/notifications/index/). | -## Create notification settings +### Create notification settings The following example request sets up notifications on a failure of a reindex task: @@ -233,5 +232,5 @@ DELETE _plugins/_im/lron/LRON:indices:data%2Fwrite%2Freindex ## Next steps -- Learn more about the [ISM API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/). +- Learn more about the [ISM API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/). - Learn more about the [Notifications]({{site.url}}{{site.baseurl}}/observing-your-data/notifications/index/) application. diff --git a/_im-plugin/notifications/dash-notifications.md b/_im-plugin/notifications/dash-notifications.md new file mode 100644 index 00000000000..590c943630f --- /dev/null +++ b/_im-plugin/notifications/dash-notifications.md @@ -0,0 +1,122 @@ +--- +layout: default +title: With OpenSearch Dashboards +parent: Index notifications +nav_order: 10 +--- + +# Setting up notifications using OpenSearch Dashboards + +You can configure global default notification settings for index operations in [Notification settings](#configuring-default-notifications). You can also configure notification settings for individual index operations. + +Notifications require the notifications plugin. See [Notifications]({{site.url}}{{site.baseurl}}/observing-your-data/notifications/index/). +{: .note} + +## Prerequisites + +All notification operations described here assume that you are on the **Index Management** page. To navigate to the **Index Management** page, do the following: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + + +## Configuring default notifications + +In the **Notification settings** interface, you can configure the default notification settings for index operations that may take longer to complete. These operations include the following: + +- Open +- Reindex +- Split +- Shrink +- Clone +- Force merge + +To configure default notifications, follow these steps: + + +1. In the **Index Management** panel, select **Notification settings**. + +1. Set up one or more notification channels as described in [Creating a notification channel](#creating-a-notification-channel). + +1. In the **Defaults for index operations** panel, select the checkboxes for all notifications you want to send. + + For each operation choice, you can separately select to be notified when the operation **Has failed** or **Has completed**. + + The operation choices are: + - **Reindex** + - **Shrink, split, clone** + - **Force merge** + - **Open** + + If you don't have permission to view notification settings, you cannot view or select the default settings. + {: .note} + + When you select a notification, the operation choice displays a **Notification channels** drop-down. + +1. For each operation choice with a selected notification, choose one or more channels from the **Notification channels** drop-down. + +1. Select **Save** in the lower right corner of the **Notification settings** page. + + +## Sending additional notifications + +You can view default notifications and send additional notifications when you perform one of the following operations: + +- Reindex +- Split +- Shrink +- Force merge + +To view default notifications and send additional notifications, follow these steps: + +1. In the **Index Management** interface, select **Indexes**. + +1. Select the index you want to perform the operation on. + +1. Select the **Actions** button in the upper right of the index page. + +1. Select the operation in the **Actions** dropdown list. For example, choose **Reindex** to perform the reindex operation. + +1. In the operation page, select the {::nomarkdown}expand icon{:/} (expand) icon next to **Advanced settings**. + + If you don't have permission to view notification settings, you will not be able to view the default settings. + {: .note} + + The **Advanced settings** panel lists the default notifications in the **Notifications** section. + +1. (Optional) To send additional notifications for this operation only, select the **Send additional notifications** checkbox. + +1. Select one or both of the **Has failed / timed out** or **Has completed** checkboxes. + +1. Select a channel from the **Notification channels** drop-down. + +1. Select the operation button in the lower right of the operation page. For example, if the operation is Reindex, select the **Reindex** button. + + +## Creating a notification channel + +To create a new notification channel, follow these steps: + +1. Select the **Manage channels** button in the upper right of the **Index Management** page. + + The notification manager opens a separate **Channels** window. + +1. In the **Channels** page, Select the **Create channel** button. + +1. In the **Name and description** panel, enter a name for the channel in the **Name** box. + +1. (Optional) In the **Description – _optional_** box, enter a description of the channel. + +1. In the **Configurations** panel, select a channel type in the **Channel type** drop-down. + + Channel settings depend on the channel type. Controls for the channel settings appear when you select a channel type. + +1. Enter settings information for the channel type you have selected. For example, an email channel requires that you select a sender type, enter a sender, and specify recipients. A Slack channel requires you to enter a Slack webhook URL. + +1. (Optional) To test the new channel, select **Send test message** in the lower left of the **Create channel** page. + +1. Select **Create** in the lower left of the **Create channel** page. + + +## Next steps + +- Learn more about the [Notifications]({{site.url}}{{site.baseurl}}/observing-your-data/notifications/index/) application. diff --git a/_im-plugin/notifications/index.md b/_im-plugin/notifications/index.md new file mode 100644 index 00000000000..91ad7235d03 --- /dev/null +++ b/_im-plugin/notifications/index.md @@ -0,0 +1,24 @@ +--- +layout: default +title: Index notifications +nav_order: 210 +has_children: true +has_toc: false +redirect_from: + - /im-plugin/notifications/ +--- + +# Notification settings + +You can configure notification settings to alert you via email or other channels of events during long-running index operations, including `open`, `reindex`, `resize`, `force merge`, and other using OpenSearch Dashboards or the Notifications API. + +See [Notifications]({{site.url}}{{site.baseurl}}/observing-your-data/notifications/index/) for more general information about the Notifications plugin. + + +## Using OpenSearch Dashboards + +Set up default notifications and extra notifications for indexes using [Notifications in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/im-plugin/notifications/dash-notifications). + +## Using the Notifications API + +Create, retrieve, update, and delete notification settings using the [Notifications API]({{site.url}}{{site.baseurl}}/im-plugin/notifications/api-notifications). diff --git a/_im-plugin/reindex-data.md b/_im-plugin/reindex/api-reindex.md similarity index 77% rename from _im-plugin/reindex-data.md rename to _im-plugin/reindex/api-reindex.md index e86d1b83c90..2fc9e1feb2d 100644 --- a/_im-plugin/reindex-data.md +++ b/_im-plugin/reindex/api-reindex.md @@ -1,47 +1,20 @@ --- layout: default -title: Reindex data -nav_order: 30 +title: Reindex API +parent: Reindex data +nav_order: 80 +redirect_from: + - /opensearch/reindex-data/ --- -# Reindex data +# Reindex API -After creating an index, you might need to make an extensive change such as adding a new field to every document or combining multiple indexes to form a new one. Rather than deleting your index, making the change offline, and then indexing your data again, you can use the `reindex` operation. +You can copy all documents from one index to another using the [Reindex Documents API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/). -With the `reindex` operation, you can copy all or a subset of documents that you select through a query to another index. Reindex is a `POST` operation. In its most basic form, you specify a source index and a destination index. - -Reindexing can be an expensive operation depending on the size of your source index. We recommend you disable replicas in your destination index by setting `number_of_replicas` to `0` and reenable them once the reindex process is complete. -{: .note } - -For a complete API reference with all parameters and advanced options, see the [Reindex Documents API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/). -{: .note} - -## How reindex works - -The reindex operation performs the following steps: - -1. **Reads documents from the source index**: OpenSearch retrieves documents from the `_source` field of the source index. -2. **Processes the documents**: Optionally applies any specified query filters, scripts, or ingest pipelines to transform the data. -3. **Writes to the destination index**: Indexes the processed documents into the destination index using the destination index's current mappings and settings. - -Note the following considerations: - -- The reindex operation reads from the `_source` field and ignores any `stored_fields` configuration. If `_source` is disabled in your source index, the reindex operation will fail. Ensure that `_source` is enabled for all documents you want to reindex. -- Documents are indexed according to the destination index's mappings, not the source index's mappings. Create your destination index with the desired mappings before reindexing. - ---- - -#### Table of contents -1. TOC -{:toc} - - ---- +The following procedures demonstrate how to reindex documents under various conditions using this API. ## Reindex all documents -You can copy all documents from one index to another. - You first need to create a destination index with your desired field mappings and settings or you can copy the ones from your source index: ```json @@ -286,4 +259,4 @@ Option | Valid values | Description | Required ## Index codec considerations -For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#reindexing). +For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/#reindexing). diff --git a/_im-plugin/reindex/dash-reindex.md b/_im-plugin/reindex/dash-reindex.md new file mode 100644 index 00000000000..cccff289dde --- /dev/null +++ b/_im-plugin/reindex/dash-reindex.md @@ -0,0 +1,91 @@ +--- +layout: default +title: Reindexing an index with OpenSearch Dashboards +parent: Reindex data +nav_order: 10 +redirect_from: + - /opensearch/reindex-data/ +--- + + +## Reindexing an index with OpenSearch Dashboards + +You can copy all documents or a subset of documents from one or more source indexes, data streams, or aliases into a destination index, data stream, or alias. The source and destination must be different. See [Reindex Documents API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/) and [Reindex Data]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/) for detailed explanations and examples. + + +### Prerequisites + +Before using the Reindex API, the following requirements and limitations must be met: + +- The _source field must enabled for all documents in the source index. +- The destination index must exist and be configured. + +Reindexing large datasets can degrade cluster performance. For more information and mitigation techniques, see [Performance optimization]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/#performance-optimization). + + +### Procedure + +To reindex an index, follow these steps: + +1. (Optional) Create a target index for the reindex operation. See [Creating an index]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops/#creating-an-index). + + You can also create an index later in the reindexing procedure by importing settings and mappings from the source index. + {: tip} + +1. In the **Index Management** panel, choose **Indexes**. + +1. Choose the **Actions** button. + +1. From the drop-down list, select **Reindex**. + +1. In the **Configure source index** panel, select one or more indexes, aliases, and data streams to reindex. + +1. Under **Specify a reindex option**, choose **Reindex all documents** or **Reindex a subset of documents**. + +1. (Optional) If you chose to reindex a subset of documents, enter a Query DSL expression in the Query expression box to select the documents you want to reindex. See [Query DSL]({{site.url}}{{site.baseurl}}/query-dsl/). + + For example, the following Query DSL filters out all documents with a timestamp after the date `2024-01-01`: + + ```json + { + "bool": { + "filter": [ + { "range": { "timestamp": { "gte": "2024-01-01" }}} + ] + } + } + +1. The most convenient way to create a target index is to import the settings and mappings from the source index. + + To create a target index, follow these steps: + + 1. In the **Configure destination index** panel, select the **Create index** button. + + 1. In the **Create index** dialog, enter a name for the index in the **Index name** box. + + 1. (Optional) In the **Index alias – _optional_** combo box, enter or select one or more aliases for the target index. + + 1. Select **Import settings and mappings**. + + 1. In the pop-up, select the source index. + + 1. (Optional) In the **Index mapping – _optional_** panel, add new fields to the target index. See [Creating an index]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops/#creating-an-index). + + 1. Select **Create**. + +1. (Optional) View advanced options by selecting {::nomarkdown}expand icon{:/} (expand) **Advanced**. + + You can set the following options in the Advanced settings: + + 1. (Optional) Select **Reindex only unique documents** to keep from duplicating documents that are already in the target index. + + 1. (Optional) Under **Version conflicts**, select Ignore conflicts during reindexing to prevent version conflicts from halting the reindex operation. + + 1. (Optional) Select **Slice this reindexing operation** to parallelize the reindexing operation by slicing it into subtasks. + + 1. (Optional) In the **Transform with ingestion pipeline – _optional_** drop-down, select an ingest pipeline to transform documents before writing them in the new index. + + 1. Select Send additional notifications to generate notifications for the reindexing operation. See [Sending additional notifications]({{site.url}}{{site.baseurl}}/im-plugin/notifications/dash-notifications#sending-additional-notifications). + +1. Select **Reindex**. + diff --git a/_im-plugin/reindex/index.md b/_im-plugin/reindex/index.md new file mode 100644 index 00000000000..15b5e83fd6d --- /dev/null +++ b/_im-plugin/reindex/index.md @@ -0,0 +1,38 @@ +--- +layout: default +title: Reindex data +nav_order: 80 +has_children: true +has_toc: false +redirect_from: + - /opensearch/reindex-data/ + - /im-plugin/reindex-data/ +--- + + +# Reindex data + +After creating an index, you might need to make an extensive change such as adding a new field to every document or combining multiple indexes to form a new one. Rather than deleting your index, making the change offline, and then indexing your data again, you can use the `reindex` operation. + +With the `reindex` operation, you can copy all or a subset of documents that you select through a query to another index. Reindex is a `POST` operation. In its most basic form, you specify a source index and a destination index. + +Reindexing can be an expensive operation depending on the size of your source index. We recommend you disable replicas in your destination index by setting `number_of_replicas` to `0` and re-enable them once the reindex process is complete. +{: .note } + +For a complete API reference with all parameters and advanced options, see the [Reindex Documents API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/). +{: .note} + + +## How reindex works + +The reindex operation performs the following steps: + +1. **Reads documents from the source index**: OpenSearch retrieves documents from the `_source` field of the source index. +2. **Processes the documents**: Optionally applies any specified query filters, scripts, or ingest pipelines to transform the data. +3. **Writes to the destination index**: Indexes the processed documents into the destination index using the destination index's current mappings and settings. + +Note the following considerations: + +- The reindex operation reads from the `_source` field and ignores any `stored_fields` configuration. If `_source` is disabled in your source index, the reindex operation will fail. Ensure that `_source` is enabled for all documents you want to reindex. +- Documents are indexed according to the destination index's mappings, not the source index's mappings. Create your destination index with the desired mappings before reindexing. + diff --git a/_im-plugin/index-templates.md b/_im-plugin/templates/api-templates.md similarity index 94% rename from _im-plugin/index-templates.md rename to _im-plugin/templates/api-templates.md index 82686f49d2a..31f0366dd85 100644 --- a/_im-plugin/index-templates.md +++ b/_im-plugin/templates/api-templates.md @@ -1,7 +1,8 @@ --- layout: default -title: Index templates -nav_order: 2 +title: Templates API +parent: Index templates +nav_order: 30 redirect_from: - /opensearch/index-templates/ --- @@ -10,7 +11,7 @@ redirect_from: Index templates let you initialize new indexes with predefined mappings and settings. For example, if you continuously index log data, you can define an index template so that all of these indexes have the same number of shards and replicas. -### Create a template +## Create a template To create an index template, use a PUT or POST request: @@ -102,7 +103,7 @@ Any additional indexes that match this pattern---`logs-2020-01-02`, `logs-2020-0 Index patterns cannot contain any of the following characters: `:`, `"`, `+`, `/`, `\`, `|`, `?`, `#`, `>`, and `<`. -### Retrieve a template +## Retrieve a template To list all index templates: @@ -129,13 +130,13 @@ To check if a specific template exists: HEAD _index_template/{name} ``` -### Configure multiple templates +## Configure multiple templates You can create multiple index templates for your indexes. If the index name matches more than one template, OpenSearch takes the mappings and settings from the template with the highest priority and applies it to the index. For example, say you have the following two templates that both match the `logs-2020-01-02` index and there’s a conflict in the `number_of_shards` field: -#### Template 1 +### Template 1 ```json PUT _index_template/template-01 @@ -153,7 +154,7 @@ PUT _index_template/template-01 } ``` -#### Template 2 +### Template 2 ```json PUT _index_template/template-02 @@ -172,7 +173,7 @@ PUT _index_template/template-02 Because `template-02` has a higher `priority` value, it takes precedence over `template-01` . The `logs-2020-01-02` index would have the `number_of_shards` value as 3 and the `number_of_replicas` as the default value 1. -### Delete a template +## Delete a template You can delete an index template using its name: @@ -180,7 +181,7 @@ You can delete an index template using its name: DELETE _index_template/daily_logs ``` -## Composable index templates +# Composable index templates Managing multiple index templates has the following challenges: @@ -194,11 +195,11 @@ You can combine component templates to compose an index template. Settings and mappings that you specify directly in the [create index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/) request override any settings or mappings specified in an index template and its component templates. {: .note } -### Create a component template +## Create a component template Let's define two component templates⁠---`component_template_1` and `component_template_2`: -#### Component template 1 +### Component template 1 ```json PUT _component_template/component_template_1 @@ -215,7 +216,7 @@ PUT _component_template/component_template_1 } ``` -#### Component template 2 +### Component template 2 ```json PUT _component_template/component_template_2 @@ -232,7 +233,7 @@ PUT _component_template/component_template_2 } ``` -### Use component templates to create an index template +## Use component templates to create an index template When creating index templates, you need to include the component templates in a `composed_of` list. @@ -283,7 +284,7 @@ PUT logs-2020-01-01 GET logs-2020-01-01 ``` -#### Example response +### Example response ```json { @@ -325,7 +326,7 @@ GET logs-2020-01-01 ``` -## Index template options +# Index template options You can specify the following template options: diff --git a/_im-plugin/templates/dash-component-templates.md b/_im-plugin/templates/dash-component-templates.md new file mode 100644 index 00000000000..43dd6a13c39 --- /dev/null +++ b/_im-plugin/templates/dash-component-templates.md @@ -0,0 +1,161 @@ +--- +layout: default +title: Using component templates +parent: Index templates +nav_order: 20 +--- + +# Using component templates +Introduced 2.7 +{: .label .label-purple } + +Component templates are reusable building blocks that encapsulate aliases, settings, and mappings. + +Use component templates to reduce the repetitive effort needed to create multiple templates with similar configurations. See [Using templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/index/). You can create a component template that defines the basic structure and settings of your indexes, then use the component template to compose templates that incorporate those elements. + +This page describes how to create and manage component templates in **OpenSearch Dashboards**. + + +## Prerequisites + +This information is for admin users who manage OpenSearch clusters. See [Creating a cluster]({{site.url}}{{site.baseurl}}/tuning-your-cluster/cluster/). + +You should be familliar with index management in OpenSearch Dashboards. See [Index Management in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/). + +All index operations described here assume that you are on the **Index Management** page. + +To navigate to the **Index Management** page, do the following: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + + +## Creating a component template + +To create a component template, follow these steps: + +1. In the **Index Management** bar, choose **Templates**. + + **Component templates** does not appear in the **Index Management** panel until you select **Templates**. + {: tip} + +1. Select **Component templates**. + +1. Choose **Create component template**. + +1. On the **Create component template** page, name the template and add alias, setting, and mapping configurations. All three configurations are optional. You can define a component template that fully defines an index, or one that defines only one or two configurations. + + To define the component template, do the following: + + 1. Name and describe the template. + + To assign the name and description, follow these steps: + + 1. In the **Name** box, enter a name. + + 1. (Optional) In the **Description – _optional_** box, enter a description. It's good practice to describe the purpose of the component template or the conditions under which it is used. + + 1. (Optional) Define aliases. To activate and define aliases for the component template, do the following: + + 1. In the **Index alias** panel, select **Use configuration**. + 1. In the **Index alias** combo box, enter or select one or more alias names (for example `current-error-log`). + + 1. (Optional) Define index settings. To activate and define settings for the component template, do the following: + + 1. In the **Index settings** panel, select **Use configuration**. + + 1. In the **Number of primary shards** combo box, select or enter the number of primary shards. You must have at least one primary shard. + 1. In the **Number of replicas** combo box, select or enter the number of replicas. + 1. In the **Refresh interval** box, enter the interval, for example `5s`. Default is 1 second (`1s`). + 1. (Optional) Expand **Advanced settings** to add additional index settings using the JSON editor. See [] + + 1. (Optional) Define index mappings. To activate and define mappings for the component template, do the following: + + 1. In the **Index mapping** panel, select **Use configuration**. + 1. Choose **Visual editor**. + + We recommend using the JSON editor only to paste an existing JSON mapping configuration. + {: .note} + + 1. Define one or more fields (typed values) and objects (containers for nested fields and objects). + + - To define a field, follow these steps: + + 1. Select **Add new field**. + 1. In the **Field name** box, enter the field name. + 1. In the **Field type** drop-down, choose the field type (for example `text`). + + - To define an object, follow these steps: + + 1. Select **Add new object**. + 1. In the **Field name** box, enter the object's name. + 1. In the **Field type** drop-down, choose `object`. + 1. To define a nested field or object, select the {::nomarkdown}plus icon{:/} (plus) icon under **Actions**. + + 1. Choose **Create component template** to save the component template. + + +## Deleting a component template + +To delete a component template, follow these steps: + +1. Select **Templates** > **Component templates**. + +1. Choose the {::nomarkdown}trashcan icon{:/} (trashcan) icon in the row of the component template you want to delete. + +1. In the **Delete** confirmation dialog, Select the **Unlink index templates and delete** checkbox. + + The component template is removed from all index templates that were constructed using the component template, subject to the following rules: + - Values determined by the component template are removed from the index templates. + - If a value determined by a component template was overridden in an index template, the override value remains. + - If a value determined by a component template was not overridden in an index template, the value is now undefined in the index template. + - Values in indexes that were created using the index templates _do not change_. Once an index is created, template changes have no effect on the index. + +1. Choose **Apply changes**. + + +## Modifying a component template + +To delete a component template, follow these steps: + +1. Select **Templates** > **Component templates**. + +1. In the **Name** column, select the component template that you want to modify. + +1. (Optional) Toggle **Use configuration** off or on for any of **Index alias**, **Index settings**, or **Index mapping** configurations. + +1. (Optional) For any of the configurations that are toggled on, you can do the following: + - Add values to the configuration. For example, you can add one or more aliases to the **Index alias** configuration. + - Remove values from the configuration. For example, you can remove the **Number of replicas** value from an **Index settings** configuration. + - Change values in the configuration. For example, you can change the **Number of replicas** value in an **Index settings** configuration. + +1. Choose **Apply changes**. + + New configurations are applied to all index templates that use the component template. + + +## Associating component templates with index templates + +Use component templates to create an index template without retyping alias, setting, or mapping configurations. + +This procedure assumes you have begun creating a new template as described in [Creating an index template]({{site.url}}{{site.baseurl}}im-plugin/templates/dash-templates/#creating-an-index-template). The procedure is the same up through [Creating the template definition]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/#creating-the-template-definition). To use component templates to construct an index template at that point, follow these steps: + +1. In the **Template settings** panel under **Choose a method to define your templates**, choose **Component templates**. + +1. In the **Component template** panel, select **Associate component template**. + +1. In the **Associate component template** dialog, select the component templates that you want to associate with your index template. + + You can select any number of component templates, including multiple component templates with the same type of configuration. If component templates have conflicting values, _the value of the last component template in the list overwrites previous values_. For example, if an index template has two component templates with index setting configurations, and both configurations define the number of primary shards, only the second configuration applies to the index template. + + For this reason, we recommend that you not create index templates that use multiple component templates with the same configuration type. For more advanced applications, such as composing mappings using two or more component templates, take care not to add conflicting configurations. + +1. Select **Associate**. + +1. (Optional) Override component template values. + + To specify individual values that override values in the component templates, follow these steps: + + 1. Select **Override template definition**. + 1. Enter or select alias, settings, or mapping values as in [Creating the template definition]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/#creating-the-template-definition). The index template will apply these values instead of the ones in the component templates. + +1. Select **Create template** (for a new template) or **Save** (if modifying a template). diff --git a/_im-plugin/templates/dash-templates.md b/_im-plugin/templates/dash-templates.md new file mode 100644 index 00000000000..d4f13cf5d29 --- /dev/null +++ b/_im-plugin/templates/dash-templates.md @@ -0,0 +1,157 @@ +--- +layout: default +title: Using templates +parent: Index templates +nav_order: 10 +--- + +# Using templates in OpenSearch Dashboards + +[Index templates]({{site.url}}{{site.baseurl}}/opensearch/index-templates/) define index mappings, settings, and aliases. Use templates to create new indexes without entering these configurations repeatedly. For example, if you continuously index log data, you can define an index template so that all of the indexes have the same number of shards and replicas. + +This page describes how to create and manage index templates and component templates in **OpenSearch Dashboards**. + + +## Prerequisites + +This information is for admin users who manage OpenSearch clusters. See [Creating a cluster]({{site.url}}{{site.baseurl}}/tuning-your-cluster/cluster/). + +You should be familliar with index management in OpenSearch Dashboards. [Index Management in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/). + +All operations described here assume that you are on the **Index Management** page. To navigate to the **Index Management** page, do the following: + +1. From the [navigation panel]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/#navigating-the-index-management-ui), select **Management** > **Index Management**. + + +## Creating an index template + +To create an index template, follow these steps: + +1. Select **Templates**. + +1. Choose **Create template**. + +1. On the **Create template** page, define the template settings and the template definition as described in the following sections. + + +### Configuring template settings + +The **Template settings** panel defines how the template is identified, including its name and the index patterns that it matches. To choose the template settings, follow these steps: + +1. In the **Template name** field, enter a name for the template. + +1. Select the **Template type**. If the template will be used to create data streams, choose **Data streams**. See [] + +1. If the template is for a data stream, enter the name of the timestamp field in the **Time field** box. + + Data streams require a timestamp field. This step is not optional if your template is for a data stream. + {: .note} + +1. In the **Index patterns** box, enter one or more index patterns that match the indexes the template applies to. See [] + + Separate the index patterns with commas. + +1. In the **Priority** box, enter the template priority. The default is 0, which is the lowest allowable priority. + + OpenSearch uses the priority in cases where an index name matches more than one template. It applies the mappings and settings from the template with the highest priority to the index. + +1. Select one of two methods to define the template: + + - To define a template from scratch, choose **Simple template**. + - To base the template on one or more component templates, choose **Component template**. See the instructions for creating a template this way in [Associating component templates with index templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-component-templates/#associating-component-templates-with-index-templates). + + The following procedures assume you've chosen **Simple template**. + + +### Creating the template definition + +In the **Template definition** panel, do the following: + + 1. In the **Index alias** combo box, enter or select one or more alias names (for example `current-log`). + +1. Under the **Index settings** heading, define the cluster parameters by doing the following: + + 1. In the **Number of primary shards** combo box, select or enter the number of primary shards. You must have at least one primary shard. + 1. In the **Number of replicas** combo box, select or enter the number of replicas. + 1. In the **Refresh interval** box, enter the interval, for example `5s`. Default is 1 second (`1s`). + 1. (Optional) Expand **Advanced settings** to add additional index settings using the JSON editor. See [] + +1. Under the **Index mapping** heading, define index fields by doing the following: + + 1. Choose **Visual editor**. + + We recommend using the JSON editor only to paste an existing JSON mapping configuration. + {: .note} + + 1. Define one or more fields (typed values) and objects (containers for nested fields and objects). + + - To define a field, follow these steps: + + 1. Select **Add new field**. + 1. In the **Field name** box, enter the field name. + 1. In the **Field type** drop-down, choose the field type (for example `text`). + + - To define an object, follow these steps: + + 1. Select **Add new object**. + 1. In the **Field name** box, enter the object's name. + 1. In the **Field type** drop-down, choose `object`. + 1. To define a nested field or object, select the {::nomarkdown}plus icon{:/} (plus) icon under **Actions**. + +1. Choose **Create template**. + + +## Deleting a template + +To delete an index template, follow these steps: + +1. Select **Templates**. + +1. Choose the {::nomarkdown}trashcan icon{:/} (trashcan) icon in the row of the template you want to delete. + +1. In the Delete Templates confirmation dialog text box, enter `delete`. + +1. Choose **Delete**. + + +## Modifying a template + +To modify an existing index template, follow these steps: + +1. Select **Templates**. + +1. In the **Template name** column, select the index template you want to modify. + +1. On the template page, choose the *Configuration* tab. + +1. Update the template configuration. Change the template parameters as described in [Configuring template settings](#configuring-template-settings) and [Configuring the template definition](#creating-the-template-definition). + +1. (Optional) To review your choices, follow these steps: + 1. Select **Preview template** in the lower left corner of the page. + 1. Review the template configuration in the **Preview template** dialog. + 1. Close the Preview template dialog by selecting **Close**. + +1. To save your changes, choose **Save** in the lower left corner of the page. + +## Applying a template + +You apply a template to an index by giving the index a name that matches one of the template's index patterns. + +To create an index using a template, follow these steps: + +1. Select **Index**. + +1. Choose **Create index**. + +1. In the **Define index** panel, enter a name name in the **Index name** box that matches an index pattern defined in the template. + + For example, if the template has an index pattern defined as `flight-data-from-templates-*`, then naming the index `flight-data-from-templates-1` applies the pattern. + + If more than one template matches the index pattern, OpenSearch applies the template with the highest priority that . + +1. (Optional) Change any alias, setting, or mapping values that you want to override from the template. + + When focus goes to another control in the Create index page, OpenSearch Dashboards applies the template. You can enter a new value and that value is saved, overriding the template value. + {: .note} + +1. At the lower right of the **Create index** page, select **Create**. diff --git a/_im-plugin/templates/index.md b/_im-plugin/templates/index.md new file mode 100644 index 00000000000..6667f38232f --- /dev/null +++ b/_im-plugin/templates/index.md @@ -0,0 +1,25 @@ +--- +layout: default +title: Index templates +nav_order: 30 +has_children: true +has_toc: false +redirect_from: + - /opensearch/index-templates/ + - /im-plugin/index-templates/ +--- + +# Index templates + +OpenSearch Dashboards | API +:-- | :-- +[Associating component templates with index templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-component-templates/#associating-component-templates-with-index-templates) | [Using component templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-component-templates/) +[Creating a component template]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-component-templates/#creating-a-component-template) | [Using component templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-component-templates/) +[Deleting a component template]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-component-templates/#deleting-a-component-template) | [Using component templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-component-templates/) +[Modifying a component template]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-component-templates/#modifying-a-component-template) | [Using component templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-component-templates/) +[Applying a template]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/#applying-a-template) | [Using templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/) +[Configuring template settings]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/#configuring-template-settings) | [Using templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/) +[Creating an index template]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/#creating-an-index-template) | [Using templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/) +[Creating the template definition]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/#creating-the-template-definition) | [Using templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/) +[Deleting a template]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/#deleting-a-template) | [Using templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/) +[Modifying a template]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/#modifying-a-template) | [Using templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/dash-templates/) diff --git a/_install-and-configure/configuring-opensearch/index-settings.md b/_install-and-configure/configuring-opensearch/index-settings.md index 4fe53eaddd6..7d3aca56155 100644 --- a/_install-and-configure/configuring-opensearch/index-settings.md +++ b/_install-and-configure/configuring-opensearch/index-settings.md @@ -173,11 +173,11 @@ OpenSearch supports the following static index-level index settings: - `qat_deflate` (OpenSearch 2.14 and later, on supported systems) - `qat_zstd` (OpenSearch 2.19.3 and later, on supported systems) -For `zstd`, `zstd_no_dict`, `qat_lz4`, `qat_deflate`, and `qat_zstd`, you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`. +For `zstd`, `zstd_no_dict`, `qat_lz4`, `qat_deflate`, and `qat_zstd`, you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/). Optional. Default is `default`. -- `index.codec.compression_level` (Integer): The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size), but slower compression and decompression speeds lead to higher indexing and search latencies. This setting can only be specified if `index.codec` is set to `zstd` or `zstd_no_dict` in OpenSearch 2.9 and later; `qat_lz4` or `qat_deflate` in OpenSearch 2.14 and later; or `qat_zstd` in OpenSearch 2.19.3 and later. Valid values are integers in the `[1, 6]` range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `3`. +- `index.codec.compression_level` (Integer): The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size), but slower compression and decompression speeds lead to higher indexing and search latencies. This setting can only be specified if `index.codec` is set to `zstd` or `zstd_no_dict` in OpenSearch 2.9 and later; `qat_lz4` or `qat_deflate` in OpenSearch 2.14 and later; or `qat_zstd` in OpenSearch 2.19.3 and later. Valid values are integers in the `[1, 6]` range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/). Optional. Default is `3`. -- `index.codec.qatmode` (String): The hardware acceleration mode used for the `qat_lz4`, `qat_deflate`, and `qat_zstd` compression codecs. Valid values are `auto` and `hardware`. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `auto` (the recommended setting). +- `index.codec.qatmode` (String): The hardware acceleration mode used for the `qat_lz4`, `qat_deflate`, and `qat_zstd` compression codecs. Valid values are `auto` and `hardware`. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/). Optional. Default is `auto` (the recommended setting). - `index.routing_partition_size` (Integer): The number of shards a custom routing value can go to. Routing helps an imbalanced cluster by relocating values to a subset of shards rather than a single shard. To enable routing, set this value to greater than 1 but less than `index.number_of_shards`. Default is 1. @@ -251,7 +251,7 @@ The following settings control the merge scheduler, which determines how merge o - `index.store.preload` (Static, list): Specifies which file extensions should be preloaded into the filesystem cache when the index is opened. This setting only works with the `mmap` directory implementation and provides best-effort caching. Preloading files can improve search performance by reducing disk I/O, but it consumes more memory. Common extensions include `nvd` (norms), `dvd` (doc values), and `tim` (terms index). Default is `[]` (empty list). -- `index.bulk.adaptive_shard_selection.enabled` (Boolean): Set to `true` to enable adaptive shard selection for bulk operations so that a single shard is chosen for append-only indexes. Default is `false`. For more information, see [Adaptive shard selection for bulk indexing]({{site.url}}{{site.baseurl}}/im-plugin/append-only-index/#adaptive-shard-selection-for-bulk-indexing). +- `index.bulk.adaptive_shard_selection.enabled` (Boolean): Set to `true` to enable adaptive shard selection for bulk operations so that a single shard is chosen for append-only indexes. Default is `false`. For more information, see [Adaptive shard selection for bulk indexing]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/append-only-index/#adaptive-shard-selection-for-bulk-indexing). ### Updating a static index setting diff --git a/_mappings/mapping-parameters/similarity.md b/_mappings/mapping-parameters/similarity.md index 8d1bd9d0903..2fb8abcb867 100644 --- a/_mappings/mapping-parameters/similarity.md +++ b/_mappings/mapping-parameters/similarity.md @@ -18,11 +18,11 @@ The `similarity` mapping parameter lets you customize how relevance scores are c OpenSearch supports two types of similarities for field mappings: **Built-in similarities** (can be used directly): -- [`BM25`]({{site.url}}{{site.baseurl}}/im-plugin/similarity/#bm25-similarity-default) (default): Uses a modern, probabilistic ranking model that balances term frequency, document length, and inverse document frequency. -- [`boolean`]({{site.url}}{{site.baseurl}}/im-plugin/similarity/#boolean-similarity): Returns constant scores (`1` or `0`), so should be used if you care only about matching, not relevance. +- [`BM25`]({{site.url}}{{site.baseurl}}im-plugin/index-other/similarity#bm25-similarity-default) (default): Uses a modern, probabilistic ranking model that balances term frequency, document length, and inverse document frequency. +- [`boolean`]({{site.url}}{{site.baseurl}}im-plugin/index-other/similarity#boolean-similarity): Returns constant scores (`1` or `0`), so should be used if you care only about matching, not relevance. **Custom similarities** (must be defined in the index settings first): -- [DFR, DFI, IB, LM Dirichlet, LM Jelinek Mercer, and scripted similarities]({{site.url}}{{site.baseurl}}/im-plugin/similarity/#available-similarity-types): Advanced similarity algorithms that require configuration in the index settings before they can be referenced by name in field mappings. +- [DFR, DFI, IB, LM Dirichlet, LM Jelinek Mercer, and scripted similarities]({{site.url}}{{site.baseurl}}im-plugin/index-other/similarity#available-similarity-types): Advanced similarity algorithms that require configuration in the index settings before they can be referenced by name in field mappings. ## Setting a custom similarity on a field @@ -98,4 +98,4 @@ You can examine the score returned in the `_score` field of the response: ## Related documentation -- [Similarity]({{site.url}}{{site.baseurl}}/im-plugin/similarity/) \ No newline at end of file +- [Similarity]({{site.url}}{{site.baseurl}}im-plugin/index-other/similarity) \ No newline at end of file diff --git a/_security/access-control/permissions.md b/_security/access-control/permissions.md index 5ee40cf283a..ce2072a86f2 100644 --- a/_security/access-control/permissions.md +++ b/_security/access-control/permissions.md @@ -216,9 +216,9 @@ Cross-references to API documentation in the permissions that follow are only in | **Permission** | **Description** | | :--- | :--- | -| `indices:admin/template/delete` | Permission to [delete index templates]({{site.url}}{{site.baseurl}}/im-plugin/index-templates/#delete-a-template). | -| `indices:admin/template/get` | Permission to [get index templates]({{site.url}}{{site.baseurl}}/im-plugin/index-templates/#retrieve-a-template). | -| `indices:admin/template/put` | Permission to [create index templates]({{site.url}}{{site.baseurl}}/im-plugin/index-templates/#create-a-template). | +| `indices:admin/template/delete` | Permission to [delete index templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/api-templates/#delete-a-template). | +| `indices:admin/template/get` | Permission to [get index templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/api-templates/#retrieve-a-template). | +| `indices:admin/template/put` | Permission to [create index templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/api-templates/#create-a-template). | | `indices:data/read/scroll` | Permission to scroll through data. This setting must be configured as both a cluster- and index-level permission. | | `indices:data/read/scroll/clear` | Permission to clear the scroll object. This setting must be configured as both a cluster- and index-level permission. | | `indices:data/read/mget` | Permission to run [multiple GET operations]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) in one request. | @@ -295,7 +295,7 @@ See [Asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/inde ### Index State Management permissions -See [ISM API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/). +See [ISM API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api-ism/). - `cluster:indices:admin/opensearch/ism/managedindex` - `cluster:admin/opendistro/ism/managedindex/add` @@ -310,7 +310,7 @@ See [ISM API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/). ### Index rollups permissions -See [Index rollups API]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/rollup-api/). +See [Index rollups API]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/api-rollup/). - `cluster:admin/opendistro/rollup/index` - `cluster:admin/opendistro/rollup/get` @@ -336,7 +336,7 @@ See [Creating reports with the Dashboards interface]({{site.url}}{{site.baseurl} ### Transform job permissions -See [Transforms APIs]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms/transforms-apis/) +See [Transforms APIs]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms/api-transform/) - `cluster:admin/opendistro/transform/index` - `cluster:admin/opendistro/transform/get` @@ -476,7 +476,7 @@ See [REST API reference]({{site.url}}{{site.baseurl}}/api-reference/index/). The index template permissions are for indexes but apply globally to the cluster. -See [Index templates]({{site.url}}{{site.baseurl}}/im-plugin/index-templates/). +See [Index templates]({{site.url}}{{site.baseurl}}/im-plugin/templates/index/). - `indices:admin/index_template/delete` - `indices:admin/index_template/get` @@ -498,13 +498,13 @@ These permissions apply to an index or index pattern. You might want a user to h | `indices:admin/close` | Permission to [close an index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/close-index/). | | `indices:admin/close*` | Permission to [close an index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/close-index/). | | `indices:admin/create` | Permission to [create indexes]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/). | -| `indices:admin/data_stream/create` | Permission to create [data streams]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/datastream/#creating-a-data-stream). | -| `indices:admin/data_stream/delete` | Permission to [delete data streams]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/datastream/#deleting-a-data-stream). | -| `indices:admin/data_stream/get` | Permission to [get data streams]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/datastream/#viewing-a-data-stream). | +| `indices:admin/data_stream/create` | Permission to create [data streams]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#creating-a-data-stream). | +| `indices:admin/data_stream/delete` | Permission to [delete data streams]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#deleting-a-data-stream). | +| `indices:admin/data_stream/get` | Permission to [get data streams]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#viewing-data-streams). | | `indices:admin/delete` | Permission to [delete indexes]({{site.url}}{{site.baseurl}}/api-reference/index-apis/delete-index/). | | `indices:admin/exists` | Permission to use [exists query]({{site.url}}{{site.baseurl}}/query-dsl/term/exists/). | -| `indices:admin/flush` | Permission to [flush an index]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index-management/#flushing-an-index). | -| `indices:admin/flush*` | Permission to [flush an index]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index-management/#flushing-an-index). | +| `indices:admin/flush` | Permission to [flush an index]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-manage/#flushing-indexes). | +| `indices:admin/flush*` | Permission to [flush an index]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-manage/#flushing-indexes). | | `indices:admin/forcemerge` | Permission to force merge indexes and data streams. | | `indices:admin/get` | Permission to get index and mapping. | | `indices:admin/mapping/put` | Permission to add new mappings and fields to an index. | @@ -519,10 +519,10 @@ These permissions apply to an index or index pattern. You might want a user to h | `indices:admin/plugins/replication/index/stop` | Permission to stop cross-cluster replication. | | `indices:admin/plugins/replication/index/update` | Permission to update cross-cluster replication settings. | | `indices:admin/plugins/replication/index/status_check` | Permission to check the status of cross-cluster replication. | -| `indices:admin/refresh` | Permission to use the [index refresh API]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index-management/#refreshing-an-index). | +| `indices:admin/refresh` | Permission to use the [index refresh API]({{site.url}}{{site.baseurl}}/im-plugin/index-ops/dash-ops/#refreshing-an-index). | | `indices:admin/refresh*` | Permission to use the index refresh API. | | `indices:admin/resolve/index` | Permission to resolve index names, index aliases and data streams. | -| `indices:admin/rollover` | Permission to perform [index rollover]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/rollover/). | +| `indices:admin/rollover` | Permission to perform [index rollover]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/dash-datastream/#rolling-over-a-data-stream). | | `indices:admin/seq_no/global_checkpoint_sync` | Permission to perform a global checkpoint sync. | | `indices:admin/settings/update` | Permission to [update index settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/update-settings/). | | `indices:admin/shards/search_shards` | Permission to perform [cross cluster search]({{site.url}}{{site.baseurl}}/security/access-control/cross-cluster-search/). | diff --git a/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md index 477a4578101..a032c476059 100644 --- a/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md +++ b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md @@ -722,7 +722,7 @@ We strongly recommend against restoring `.opendistro_security` using an admin ce ## Index codec considerations -For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#snapshots). +For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/#snapshots). ## Related documentation diff --git a/_tuning-your-cluster/performance.md b/_tuning-your-cluster/performance.md index 8ec9f9a553c..3b74a84d1bf 100644 --- a/_tuning-your-cluster/performance.md +++ b/_tuning-your-cluster/performance.md @@ -140,4 +140,4 @@ POST /_bulk?pretty&filter_path=-took,-items.index._index,-items.index._type ## Compression codecs -In OpenSearch 2.9 and later, there are two new codecs for compression: `zstd` and `zstd_no_dict`. You can optionally specify a compression level for these in the `index.codec.compression_level` setting with values in the [1, 6] range. [Benchmark]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#benchmarking) data shows that `zstd` provides a 7% better write throughput and `zstd_no_dict` provides a 14% better throughput, along with a 30% improvement in storage compared with the `default` codec. For more information about compression, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). +In OpenSearch 2.9 and later, there are two new codecs for compression: `zstd` and `zstd_no_dict`. You can optionally specify a compression level for these in the `index.codec.compression_level` setting with values in the [1, 6] range. [Benchmark]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/#benchmarking) data shows that `zstd` provides a 7% better write throughput and `zstd_no_dict` provides a 14% better throughput, along with a 30% improvement in storage compared with the `default` codec. For more information about compression, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-other/index-codecs/). diff --git a/images/admin-ui-index/component.gif b/images/admin-ui-index/component.gif deleted file mode 100644 index 9c3950b264d..00000000000 Binary files a/images/admin-ui-index/component.gif and /dev/null differ diff --git a/images/dashboards/index-management-UI-callouts.png b/images/dashboards/index-management-UI-callouts.png new file mode 100644 index 00000000000..17e41c28b41 Binary files /dev/null and b/images/dashboards/index-management-UI-callouts.png differ diff --git a/images/dashboards/index-management-UI.png b/images/dashboards/index-management-UI.png index b9b6cc2e919..189009e5a57 100644 Binary files a/images/dashboards/index-management-UI.png and b/images/dashboards/index-management-UI.png differ