feat: Add lifespan OpenSearch bootstrap via service JWT

[edwinjosechittilappilly]

This pull request introduces a new mechanism for securely bootstrapping OpenSearch security using a platform-issued JWT at application startup, and adds a utility for verifying JWTs using issuer-hosted public keys. It also includes supporting refactors and comprehensive unit tests for the new JWT verification logic.

OpenSearch Security Bootstrap Enhancements:

• Added a new startup flow in lifespan.py to perform a one-shot OpenSearch security bootstrap using an admin username derived from a platform-issued service JWT (OPENRAG_SERVICE_TOKEN). This is controlled by the new OPENRAG_BOOTSTRAP_OS_SECURITY_ON_STARTUP setting, ensuring the admin role mapping is in place before other startup tasks. The corresponding call in startup_tasks is suppressed when this flag is enabled. [1] [2] [3] [4] [5]

• Added get_openrag_service_token accessor and related environment variable handling in settings.py to support dynamic retrieval of the service token.

• Refactored OpenSearch client creation: renamed and generalized the method to create_opensearch_client_from_jwt, and made create_user_opensearch_client a wrapper. [1] [2]

JWT Verification Utility:

• Added config/utils.py with robust utilities for fetching and caching public keys from JWT issuers (supporting PEM, JWKS, and JWK formats), and for verifying JWTs by dynamically discovering the signing key from the token's iss claim. This allows for flexible, issuer-driven JWT verification suitable for platform-managed deployments.

• Added admin_username_from_service_jwt in auth/ibm_auth.py to extract the admin username from a platform-issued service JWT, matching the precedence logic used elsewhere.

Testing Improvements:

• Added comprehensive unit tests for the new JWT issuer verification logic, covering PEM, JWKS, and raw PEM key responses.

• Minor test setup improvements for isolation in test_jwt_claims_cache.py. [1] [2]

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• ✅ Review completed - (🔄 Check again to review again)

Walkthrough

Adds a lifespan-startup OpenSearch security bootstrap gated by OPENRAG_BOOTSTRAP_OS_SECURITY_ON_STARTUP and PLATFORM_SERVICE_JWT: the app derives an admin username from a service JWT, creates a temporary JWT-authenticated OpenSearch client, waits for readiness, runs setup_opensearch_security, and the orchestrator skips duplicate setup.

Changes

OpenSearch Bootstrap Startup Flow

Layer / File(s)	Summary
Bootstrap configuration, service-token helpers, and client factory src/config/settings.py, src/config/utils.py	Adds K8S_SA_TOKEN_PATH, a K8s SA token reader, get_opensearch_service_token(), module-level OPENRAG_SERVICE_TOKEN/PLATFORM_SERVICE_JWT, OPENRAG_BOOTSTRAP_OS_SECURITY_ON_STARTUP, and AppClients.create_opensearch_client_from_jwt() with create_user_opensearch_client() delegating to it.
JWT admin username extraction src/auth/ibm_auth.py	Adds admin_username_from_service_jwt() which decodes the platform service JWT without signature verification and returns username or sub, or None on decode failure.
Lifespan startup bootstrap src/app/lifespan.py	Implements conditional bootstrap in run_startup() that validates PLATFORM_SERVICE_JWT, derives admin username, creates a temporary OpenSearch client from the JWT, waits for readiness, calls setup_opensearch_security(admin_username), and closes the client.
Orchestrator skip logic integration src/services/startup_orchestrator.py	Imports the bootstrap flag and updates startup branching to skip the standard OpenSearch security setup when lifespan bootstrap mode is enabled, avoiding duplicate configuration.

Sequence Diagram

sequenceDiagram
  participant Config
  participant KubeSA
  participant AuthServer
  participant Lifespan
  participant IBMAuth
  participant OpenSearch
  Config->>KubeSA: read K8S_SA_TOKEN (K8S_SA_TOKEN_PATH)
  KubeSA-->>Config: K8S_SA_TOKEN
  Config->>AuthServer: POST /internal/token/opensearch {tenant_id} with Bearer K8S_SA_TOKEN
  AuthServer-->>Config: { token: OPENRAG_SERVICE_TOKEN }
  Lifespan->>IBMAuth: admin_username_from_service_jwt(PLATFORM_SERVICE_JWT)
  Lifespan->>OpenSearch: create temporary client from PLATFORM_SERVICE_JWT
  Lifespan->>OpenSearch: wait_for_opensearch()
  Lifespan->>OpenSearch: setup_opensearch_security(admin_username)
  OpenSearch-->>Lifespan: bootstrap complete
  Lifespan->>OpenSearch: close temporary client

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• langflow-ai/openrag#1603: Both PRs update startup gating for OpenSearch security setup; this PR adds OPENRAG_BOOTSTRAP_OS_SECURITY_ON_STARTUP handling while the other introduces OPENRAG_SKIP_OS_SECURITY_SETUP.

Suggested reviewers

• zzzming

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'feat: Add lifespan OpenSearch bootstrap via service JWT' accurately and specifically describes the main change: adding an OpenSearch security bootstrap mechanism during FastAPI lifespan using service JWT authentication.
Docstring Coverage	✅ Passed	Docstring coverage is 90.91% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat-add-os-setup-bootstrap

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/auth/ibm_auth.py`:
- Around line 35-49: admin_username_from_service_jwt currently returns any
truthy username or sub claim which may be non-string; update the function to
validate that the chosen claim is a string before returning it (if not a string,
return None). Locate admin_username_from_service_jwt, after decoding and
selecting claims.get("username") or claims.get("sub"), check isinstance(value,
str) and only return it when true; optionally log a warning when a non-string
claim is encountered to aid debugging.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a9186d8d-645a-464d-9299-7a929ec75224

📥 Commits

Reviewing files that changed from the base of the PR and between a4b2da8 and 3be0c7a.

📒 Files selected for processing (4)

• src/app/lifespan.py
• src/auth/ibm_auth.py
• src/config/settings.py
• src/services/startup_orchestrator.py