Help with Backup and Restoring Reflex OpenSearch Database

[greycel]

Hi Team,

We are trying to create another instance of reflex-soar with an existing database.

• Copied all the "reflex-*" tables from the existing OpenSearch cluster to a new cluster using elasticdump, but the API service is failing with the below error.

• New tables are being created by the API service alongside the ones that were copied using elasticdump, like for example having a table with version "reflex-users-0.1.5" that was copied from the existing cluster to a new cluster and after initiating API service a new table "reflex-users" is getting created on the new cluster.

Is there any document or script that can help us out with this?

reflex-api02  | 2024-05-18 04:44:21,218 - app - INFO - Updating index template for reflex-case-comments
reflex-api02  | 2024-05-18 04:44:21,219 - app - INFO - Creating index reflex-threat-values-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-case-history
reflex-api02  | 2024-05-18 04:44:21,228 - app - INFO - Updating index template for reflex-case-history
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-case-comments-0.1.4
reflex-api02  | 2024-05-18 04:44:21,238 - app - INFO - Creating index reflex-case-comments-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-cases
reflex-api02  | 2024-05-18 04:44:21,246 - app - INFO - Updating index template for reflex-cases
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-case-tasks
reflex-api02  | 2024-05-18 04:44:21,246 - app - INFO - Updating index template for reflex-case-tasks
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-event-rules-0.1.5
reflex-api02  | 2024-05-18 04:44:21,247 - app - INFO - Creating index reflex-event-rules-0.1.5
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-case-history-0.1.4
reflex-api02  | 2024-05-18 04:44:21,252 - app - INFO - Creating index reflex-case-history-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-case-templates
reflex-api02  | 2024-05-18 04:44:21,258 - app - INFO - Updating index template for reflex-case-templates
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-observables-test
reflex-api02  | 2024-05-18 04:44:21,259 - app - INFO - Updating index template for reflex-observables-test
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-cases-0.1.4
reflex-api02  | 2024-05-18 04:44:21,270 - app - INFO - Creating index reflex-cases-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-case-tasks-0.1.4
reflex-api02  | 2024-05-18 04:44:21,278 - app - INFO - Creating index reflex-case-tasks-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-agent-groups
reflex-api02  | 2024-05-18 04:44:21,280 - app - INFO - Updating index template for reflex-agent-groups
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-agent-policies
reflex-api02  | 2024-05-18 04:44:21,282 - app - INFO - Updating index template for reflex-agent-policies
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-case-task-notes
reflex-api02  | 2024-05-18 04:44:21,283 - app - INFO - Updating index template for reflex-case-task-notes
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-plugins
reflex-api02  | 2024-05-18 04:44:21,285 - app - INFO - Updating index template for reflex-plugins
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-agent-policies-0.1.4
reflex-api02  | 2024-05-18 04:44:21,299 - app - INFO - Creating index reflex-agent-policies-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-plugin-configs
reflex-api02  | 2024-05-18 04:44:21,306 - app - INFO - Updating index template for reflex-plugin-configs
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-case-task-notes-0.1.4
reflex-api02  | 2024-05-18 04:44:21,309 - app - INFO - Creating index reflex-case-task-notes-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-audit-logs
reflex-api02  | 2024-05-18 04:44:21,310 - app - INFO - Updating index template for reflex-audit-logs
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-users
reflex-api02  | 2024-05-18 04:44:21,314 - app - INFO - Updating index template for reflex-users
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-user-roles
reflex-api02  | 2024-05-18 04:44:21,316 - app - INFO - Updating index template for reflex-user-roles
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-audit-logs-0.1.4
reflex-api02  | 2024-05-18 04:44:21,329 - app - INFO - Creating index reflex-audit-logs-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-data-types
reflex-api02  | 2024-05-18 04:44:21,330 - app - INFO - Updating index template for reflex-data-types
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-case-statuses
reflex-api02  | 2024-05-18 04:44:21,335 - app - INFO - Updating index template for reflex-case-statuses
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-close-reasons
reflex-api02  | 2024-05-18 04:44:21,348 - app - INFO - Updating index template for reflex-close-reasons
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-settings
reflex-api02  | 2024-05-18 04:44:21,352 - app - INFO - Updating index template for reflex-settings
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-inputs
reflex-api02  | 2024-05-18 04:44:21,359 - app - INFO - Updating index template for reflex-inputs
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-organizations
reflex-api02  | 2024-05-18 04:44:21,364 - app - INFO - Updating index template for reflex-organizations
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-inputs-0.1.4
reflex-api02  | 2024-05-18 04:44:21,376 - app - INFO - Creating index reflex-inputs-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-observable-history
reflex-api02  | 2024-05-18 04:44:21,377 - app - INFO - Updating index template for reflex-observable-history
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-tasks
reflex-api02  | 2024-05-18 04:44:21,384 - app - INFO - Updating index template for reflex-tasks
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-detections
reflex-api02  | 2024-05-18 04:44:21,386 - app - INFO - Updating index template for reflex-detections
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-detections-log
reflex-api02  | 2024-05-18 04:44:21,392 - app - INFO - Updating index template for reflex-detections-log
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-observable-history-0.1.4
reflex-api02  | 2024-05-18 04:44:21,398 - app - INFO - Creating index reflex-observable-history-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-mitre-tactics
reflex-api02  | 2024-05-18 04:44:21,402 - app - INFO - Updating index template for reflex-mitre-tactics
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-tasks-0.1.4
reflex-api02  | 2024-05-18 04:44:21,405 - app - INFO - Creating index reflex-tasks-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-mitre-techniques
reflex-api02  | 2024-05-18 04:44:21,411 - app - INFO - Updating index template for reflex-mitre-techniques
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-event-views
reflex-api02  | 2024-05-18 04:44:21,418 - app - INFO - Updating index template for reflex-event-views
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-mitre-tactics-0.1.4
reflex-api02  | 2024-05-18 04:44:21,423 - app - INFO - Creating index reflex-mitre-tactics-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-notification-channels
reflex-api02  | 2024-05-18 04:44:21,426 - app - INFO - Updating index template for reflex-notification-channels
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-notifications
reflex-api02  | 2024-05-18 04:44:21,429 - app - INFO - Updating index template for reflex-notifications
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-mitre-techniques-0.1.5
reflex-api02  | 2024-05-18 04:44:21,433 - app - INFO - Creating index reflex-mitre-techniques-0.1.5
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-field-mapping-templates
reflex-api02  | 2024-05-18 04:44:21,439 - app - INFO - Updating index template for reflex-field-mapping-templates
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-agent-logs
reflex-api02  | 2024-05-18 04:44:21,441 - app - INFO - Updating index template for reflex-agent-logs
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-notification-channels-0.1.4
reflex-api02  | 2024-05-18 04:44:21,448 - app - INFO - Creating index reflex-notification-channels-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-notifications-0.1.4
reflex-api02  | 2024-05-18 04:44:21,449 - app - INFO - Creating index reflex-notifications-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-email-notification-templates
reflex-api02  | 2024-05-18 04:44:21,454 - app - INFO - Updating index template for reflex-email-notification-templates
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-service-accounts
reflex-api02  | 2024-05-18 04:44:21,455 - app - INFO - Updating index template for reflex-service-accounts
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-assets
reflex-api02  | 2024-05-18 04:44:21,471 - app - INFO - Updating index template for reflex-assets
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-detection-repositories
reflex-api02  | 2024-05-18 04:44:21,475 - app - INFO - Updating index template for reflex-detection-repositories
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-detection-access-tokens
reflex-api02  | 2024-05-18 04:44:21,480 - app - INFO - Updating index template for reflex-detection-access-tokens
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-detection-repository-subscriptions
reflex-api02  | 2024-05-18 04:44:21,483 - app - INFO - Updating index template for reflex-detection-repository-subscriptions
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-detections-state
reflex-api02  | 2024-05-18 04:44:21,494 - app - INFO - Updating index template for reflex-detections-state
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-repository-sync-log
reflex-api02  | 2024-05-18 04:44:21,511 - app - INFO - Updating index template for reflex-repository-sync-log
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-integrations
reflex-api02  | 2024-05-18 04:44:21,519 - app - INFO - Updating index template for reflex-integrations
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-detections-state-0.1.4
reflex-api02  | 2024-05-18 04:44:21,521 - app - INFO - Creating index reflex-detections-state-0.1.4
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-integration-configurations
reflex-api02  | 2024-05-18 04:44:21,522 - app - INFO - Updating index template for reflex-integration-configurations
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-integration-logs
reflex-api02  | 2024-05-18 04:44:21,529 - app - INFO - Updating index template for reflex-integration-logs
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-integration-action-queue
reflex-api02  | 2024-05-18 04:44:21,534 - app - INFO - Updating index template for reflex-integration-action-queue
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-sso-providers
reflex-api02  | 2024-05-18 04:44:21,542 - app - INFO - Updating index template for reflex-sso-providers
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-sso-role-mapping-policies
reflex-api02  | 2024-05-18 04:44:21,555 - app - INFO - Updating index template for reflex-sso-role-mapping-policies
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-packages
reflex-api02  | 2024-05-18 04:44:21,558 - app - INFO - Updating index template for reflex-packages
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-data-source-templates
reflex-api02  | 2024-05-18 04:44:21,564 - app - INFO - Updating index template for reflex-data-source-templates
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Updating index template for reflex-schedules
reflex-api02  | 2024-05-18 04:44:21,571 - app - INFO - Updating index template for reflex-schedules
reflex-api02  | [2024-05-18 04:44:21 +0000] [1] [INFO] Creating index reflex-schedules-0.1.4
reflex-api02  | 2024-05-18 04:44:21,593 - app - INFO - Creating index reflex-schedules-0.1.4
reflex-api02  | False positive missing from Default Organization - {'title': 'False positive', 'description': 'Event matched detection rule but is not malicious', 'enabled': True}
reflex-api02  | No action required missing from Default Organization - {'title': 'No action required', 'description': 'No action required', 'enabled': True}
reflex-api02  | True positive missing from Default Organization - {'title': 'True positive', 'description': 'Event is malicious', 'enabled': True}
reflex-api02  | Other missing from Default Organization - {'title': 'Other', 'description': 'Any other reason not listed', 'enabled': True}
reflex-api02  | Insufficient Information missing from Default Organization - {'title': 'Insufficient Information', 'description': 'Additional enrichment and data is needed for this alert to be actionable.', 'enabled': True}
reflex-api02  | Informational Event missing from Default Organization - {'title': 'Informational Event', 'description': 'Detection provides data that is not normally malicious but should be evaluated to ensure it is expected.', 'enabled': True}
reflex-api02  | Rule Defective missing from Default Organization - {'title': 'Rule Defective', 'description': 'Alert rule is not firing correctly.', 'enabled': True}
reflex-api02  | Benign Activity missing from Default Organization - {'title': 'Benign Activity', 'description': 'Event is not malicious', 'enabled': True}
reflex-api02  | default_case_created missing from Default Organization - EmailNotificationTemplate(index='reflex-email-notification-templates', id='OfcDio8BzbVuEgSS5FgK')
reflex-api02  | Traceback (most recent call last):
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/bin/gunicorn", line 8, in <module>
reflex-api02  |     sys.exit(run())
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
reflex-api02  |     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/gunicorn/app/base.py", line 236, in run
reflex-api02  |     super().run()
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
reflex-api02  |     Arbiter(self).run()
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/gunicorn/arbiter.py", line 58, in __init__
reflex-api02  |     self.setup(app)
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/gunicorn/arbiter.py", line 118, in setup
reflex-api02  |     self.app.wsgi()
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
reflex-api02  |     self.callable = self.load()
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
reflex-api02  |     return self.load_wsgiapp()
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
reflex-api02  |     return util.import_app(self.app_uri)
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/gunicorn/util.py", line 424, in import_app
reflex-api02  |     app = app(*args, **kwargs)
reflex-api02  |   File "/app/__init__.py", line 300, in create_app
reflex-api02  |     setup(app, check_for_default=True)
reflex-api02  |   File "/app/__init__.py", line 229, in setup
reflex-api02  |     initial_settings(Settings, org_id=None, check_for_default=check_for_default)
reflex-api02  |   File "/app/defaults.py", line 979, in initial_settings
reflex-api02  |     existing_settings = cls.search().filter(
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/opensearch_dsl/response/__init__.py", line 45, in __getitem__
reflex-api02  |     return self.hits[key]
reflex-api02  |   File "/root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.8/site-packages/opensearch_dsl/utils.py", line 93, in __getitem__
reflex-api02  |     l = self._l_[k]
reflex-api02  | IndexError: list index out of range
reflex-api02 exited with code 0

[n3tsurge]

Reflex make a heavy use of index aliases and index templates and if the alias/templates arent created and pointing to the correct index it will create an index with that name instead of a versioned index name.

Also ensure you are using the exact same version of API in both systems.

To fully restore or migrate:

1. Stop the Reflex API you started that's making unversioned index names
2. In the new target Opensearch cluster you should remove all indices
3. set the environment variable REFLEX_RECOVERY_MODE=true
4. start Reflex API, let it create all indices
5. shut down reflex api
6. delete all reflex indices
7. restore or migrate your data ensuring index names are correct
8. Remove the env var
9. Do a test search in Opensearch to ensure the index alias is actually working right
10. Restart API

If this doesn't work this is something we would need support agreement to troubleshoot further.

[greycel]

Thank you, will try these steps.