feat: support cloud OpenSearch via environment variables

[fudongyingluck]

• Make OpenSearch Dashboards URL configurable (host, port, protocol)
• Move opensearch and opensearch-dashboards to docker-compose.local-opensearch.yml, included by default; comment out INCLUDE_COMPOSE_LOCAL_OPENSEARCH in .env to switch to a cloud cluster
• Make data-prepper pipelines use OPENSEARCH_HOST/PORT placeholders instead of hardcoded opensearch:9200

Description

Previously, OpenSearch and OpenSearch Dashboards were always started as local Docker containers with hardcoded hostnames (opensearch, opensearch-dashboards) and the data-prepper pipelines had a hardcoded opensearch:9200 endpoint.

This change makes the stack work with both local and cloud-hosted OpenSearch:

• Moves the opensearch and opensearch-dashboards services into a new docker-compose.local-opensearch.yml, included by default via INCLUDE_COMPOSE_LOCAL_OPENSEARCH in .env. To use a cloud cluster, users simply comment out that variable and set OPENSEARCH_HOST, OPENSEARCH_DASHBOARDS_HOST, and OPENSEARCH_DASHBOARDS_PROTOCOL.

• Adds OPENSEARCH_DASHBOARDS_HOST and OPENSEARCH_DASHBOARDS_PROTOCOL env vars (defaults: opensearch-dashboards, http) so the init container can reach both local and remote Dashboards instances.

• Replaces hardcoded opensearch:9200 in data-prepper pipeline templates with OPENSEARCH_HOST/OPENSEARCH_PORT placeholders, consistent with how credentials were already handled.

  • The opensearch-dashboards-init script already polls for Dashboards availability, so the depends_on health check
    is no longer needed.

  Default behaviour is unchanged for users running the full local stack.

Thanks for your time and any ideas

[kylehounslow]

Thanks for this contribution @fudongyingluck! Supporting cloud-hosted OpenSearch is something we've been discussing — the approach of extracting local services into a separate compose file via include: is clean, and the depends_on override pattern to re-attach health check dependencies in local mode is well done.

I've gone through the changes in detail and have some questions and concerns before we can merge.

---

1. Default local install verification

Our primary user entrypoint is:

curl -fsSL https://raw.githubusercontent.com/opensearch-project/observability-stack/main/install.sh | bash

Since the PR changes which compose files are included by default, could you run ./install.sh locally against your branch and confirm:

1. A fresh docker compose up -d starts all services including opensearch and opensearch-dashboards from the new docker-compose.local-opensearch.yml
2. OpenSearch is healthy on localhost:9200 and Dashboards on localhost:5601
3. The init container (opensearch-dashboards-init) completes successfully
4. Data Prepper and OTel Collector connect to OpenSearch without errors

2. logging: config lost on moved services

The original opensearch and opensearch-dashboards services in docker-compose.yml used logging: *logging which references the x-default-logging YAML anchor (json-file driver, 5MB max, 2 files). Since YAML anchors don't cross include: file boundaries, the moved services in docker-compose.local-opensearch.yml lose their log rotation config. I verified this with finch compose config — the merged opensearch service has no logging: section while services that stayed in the main file (like otel-collector) still have it.

These are the two biggest disk consumers in the stack, so unbounded logging could be a problem. Could you either define the logging config inline in the new file, or add the x-default-logging extension there?

3. opensearch_dashboards.yml still hardcoded

docker-compose/opensearch-dashboards/opensearch_dashboards.yml line 12 still has:

opensearch.hosts: ["https://opensearch:9200"]

This static config file is mounted into the Dashboards container. In many OSD versions, the mounted config file takes precedence over the OPENSEARCH_HOSTS environment variable. Have you tested that the env var actually overrides this when pointing to a cloud cluster? We may need to templatize this file similar to how pipelines.template.yaml works, or use sed substitution at startup.

4. OPENSEARCH_PROTOCOL used but not defined

The init script references os.getenv("OPENSEARCH_PROTOCOL", "https") but OPENSEARCH_PROTOCOL is never added to .env. You added OPENSEARCH_DASHBOARDS_PROTOCOL — could you add OPENSEARCH_PROTOCOL alongside it for consistency? Cloud clusters behind a load balancer might use http.

5. verify=False added to Dashboards health check

Adding verify=False to the wait_for_dashboards() request is fine for local dev (self-signed certs), but for cloud mode with real TLS this silently disables certificate verification. Not a blocker, but worth a code comment noting why it's there.

6. Documentation

Could you add a section to the README explaining how to switch to cloud mode? Something like:

1. Comment out INCLUDE_COMPOSE_LOCAL_OPENSEARCH in .env
2. Set OPENSEARCH_HOST, OPENSEARCH_PORT, OPENSEARCH_DASHBOARDS_HOST, OPENSEARCH_DASHBOARDS_PORT, OPENSEARCH_DASHBOARDS_PROTOCOL
3. Any caveats (TLS, auth, network reachability)

Thank you!

[fudongyingluck]

@kylehounslow
Thank you so much for the thorough review! I also want to apologize for the number of issues in the previous submission — we hadn't tested the scenario of using a cloud-hosted OpenSearch with a local OpenSearch Dashboards instance, which led to several gaps being overlooked.

One small thing I'd like to discuss before updating the PR: I noticed that install.sh currently doesn't wait for the opensearch-dashboards-init container to finish before declaring the installation complete. This init container handles workspace creation, index patterns, and saved queries — so if the user starts exploring Dashboards immediately after install.sh exits, those resources may not be set up yet.

Do you think we should add a wait step for the init container? My instinct is yes, but I wanted to get your thoughts on whether the added wait time is worth it, or if it's acceptable to let it run in the background.

[fudongyingluck]

To better support different user needs and testing scenarios, I've split the OpenSearch and OpenSearch Dashboards services into two separate compose files, so users can choose to include either or both independently.

I tested three scenarios via install.sh:

• Both OpenSearch and OpenSearch Dashboards running locally
• OpenSearch on a cloud cluster, OpenSearch Dashboards running locally
• Both OpenSearch and OpenSearch Dashboards on cloud

All three scenarios work end-to-end.

Thanks for taking the time to review — looking forward to any feedback you have!