As a developer, I want to run the integration tests locally and have github actions do it

[al-niessner]

Checked for duplicates

No - I haven't checked

🧑‍🔬 User Persona(s)

As a registry-api developer, I need the integration test suite to run automatically in GitHub Actions on every PR and push, so that I get immediate, reliable pass/fail feedback without needing a local environment.

Integration tests were previously removed from GitHub Actions because the standard ubuntu-latest runner (7GB RAM) runs out of memory when the full stack (OpenSearch + registry-api + reg-loader + reg-sweepers + Newman) runs together. The root cause is that OpenSearch's JVM is configured with -XX:-UseContainerSupport, which disables container-aware memory sizing and causes the JVM to allocate heap as a fraction of the physical host RAM, exhausting the runner.

Self-hosted runners on AWS are a non-starter due to maintenance burden and availability concerns.

💪 Motivation

• Developers should be able to validate changes via GitHub Actions before merging, without requiring a local docker-compose environment.
• Branch matching between registry-api and registry repos should be leveraged so integration tests automatically use the correct version of each component.
• As the test suite grows, the solution must remain viable — memory pressure will only increase over time.

📖 Additional Details

Current failure mode: GitHub Actions runner OOM-kills the OpenSearch container during the integration test run, causing the workflow to hang or fail silently.

Options investigated (see detailed analysis in comments):

1. Fix JVM + memory limits in docker-compose — Replace -XX:-UseContainerSupport with explicit heap bounds (OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m) and add container mem_limit. Free, lowest risk, should be tried first.
2. GitHub larger hosted runners — Change runs-on to a 4-core/16GB runner (~$0.16/run). One-line YAML change, works within GitHub Actions natively.
3. Amazon OpenSearch Serverless (AOSS) — Provision a per-run AOSS collection (keyed to GitHub SHA), run tests against it, then tear it down. Offloads OpenSearch memory entirely from the runner. Optimal long-term but requires OIDC auth setup and test harness refactoring (~$0.16/run).
4. Always-on AWS OpenSearch Service — Small persistent domain for CI only (~$25-30/month). Simpler to integrate than AOSS but introduces shared state between concurrent runs.

Acceptance Criteria

Given a developer opens or updates a PR against any branch in registry-api
When the PR is created or a new commit is pushed
Then GitHub Actions automatically triggers the integration test suite
And the pass/fail result is visible as a status check directly on the PR

---

Given a developer wants to run the same integration tests locally
When they follow the documented local testing steps using docker-compose from the registry repo
Then the tests execute successfully using branch-matched images of registry-api and registry

---

Given the integration test suite grows over time
When new tests are added
Then the CI solution continues to function without requiring runner memory increases or architectural rework (i.e., the solution scales with a growing test suite)

⚙️ Engineering Details

Recommended implementation path:

1. First, attempt Option 1 (JVM/memory limits in docker-compose in the registry repo) — zero cost, may fully resolve the issue.
2. If Option 1 is insufficient, implement Option 2 (GitHub larger runner) as a fast follow — single-line change in the workflow.
3. Track Option 3 (AOSS) as the long-term target if test suite memory footprint continues to grow.

The workflow should live in this repo (.github/workflows/) and invoke the integration test profile from the registry repo docker-compose, using branch-matching logic to select the correct registry-api image.

🎉 I&T

No response

[al-niessner]

Using per minute larger runners are integration tests take about 10 minutes to run on a semi-capable machine. At no time is it CPU or memory bound. Suspect a delay, but may just be docker compose waiting for all of the services to run rather than a shell sleep - they look the same from the outside.

Optimal solution is to move the opensearch DB to AOSS. Create it based on github sha then delete it when done. All CPUish stuff is done on github action push the bulk of the memory (opesearch) to AOSS. Could do the same with self hosted, but the availability of AOSS is exceptionally high compared to self hosting. However, hitting run-again on the github action step is not hard.

[jordanpadams]

@al-niessner one question: what exactly is the bottleneck in terms of memory and time? is it all the data we are trying to load? is the queries themselves? is it just setting up the docker instance? is it running other non-registry-related tests as well>

[jordanpadams]

Detailed Analysis & Options

Root Cause

The current docker-compose sets -XX:-UseContainerSupport on OpenSearch, which disables container-aware JVM memory sizing. Without this flag, the JVM allocates heap as a fraction of the physical host RAM — on a 7GB GitHub runner, OpenSearch's JVM will claim 1.5–2GB of heap by default. Combined with the API service, reg-loader, reg-sweepers, nginx, and the OS itself, total memory easily exceeds 7GB.

---

Options

Option 1: Fix JVM memory limits in docker-compose (free, fastest)

The quickest fix — no infrastructure changes needed:

• Replace -XX:-UseContainerSupport with explicit heap bounds: OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
• Add mem_limit: 1g on the OpenSearch container to enforce a hard cap
• Disable the OpenSearch security plugin for CI (plugins.security.disabled=true) — this also eliminates the cert generation step, simplifying the workflow
• Set JVM heap limits on any other Java services in the stack

This alone may be sufficient to fit within the 7GB runner limit. Low risk since this is test-only configuration. Should be prototyped first before investing in infrastructure changes.

Effort: Low | Cost: Free | Risk: Low

---

Option 2: GitHub-hosted larger runners (easiest infra option)

GitHub offers larger hosted runners with more RAM:

Runner	RAM	Est. cost/10-min run
ubuntu-latest (current)	7GB	$0.08
ubuntu-latest-4-cores	16GB	~$0.16
ubuntu-latest-8-cores	32GB	~$0.32

Change is a single line in the workflow: runs-on: ubuntu-latest-4-cores

Requires: GitHub Enterprise Cloud or a billing-configured org (NASA-PDS may already qualify under GitHub's nonprofit/open-source programs.

Effort: Minimal | Cost: ~/bin/bash.16/run | Risk: Very low

---

Option 3: Amazon OpenSearch Serverless (AOSS) — optimal long-term

As noted in the prior comment, this is the most scalable approach:

1. Use GitHub OIDC → AWS to authenticate without stored credentials
2. On workflow start: create an AOSS collection named after the GitHub SHA
3. Run integration tests pointing at AOSS instead of a local docker container
4. On workflow end: delete the collection

OpenSearch runs in AWS; the GitHub runner only needs to execute the API service, reg-loader, and Newman — well within memory limits.

Cost: AOSS minimum is 2 OCUs (search) + 2 OCUs (index) = ~/bin/bash.96/hr → ~/bin/bash.16 per 10-minute run. Comparable to Option 2 but scales better as test suite grows.

Effort: High (requires refactoring docker-compose / test harness to support remote OpenSearch target) | Cost: ~/bin/bash.16/run | Risk: Medium (new infra dependency)

---

Option 4: AWS-hosted OpenSearch Service (t3.small, always-on)

Provision a small persistent domain in AWS used exclusively for CI:

• Always available, no spin-up delay
• Tests point at the shared domain; each run loads/clears test data
• ~5–30/month fixed cost regardless of run frequency

Simpler than AOSS to integrate with existing docker-compose (just swap the OpenSearch endpoint), but introduces shared mutable state between concurrent CI runs.

Effort: Medium | Cost: ~5–30/month | Risk: Medium (concurrent run conflicts)

---

Recommended Path

1. Immediately: Try Option 1 (JVM + memory limits in docker-compose) — zero cost and may resolve the issue entirely.
2. If Option 1 insufficient: Option 2 (larger GitHub runner) is one line of YAML and buys headroom as the test suite grows.
3. Long-term: Option 3 (AOSS) is the right architecture if the test suite continues to grow, since runner memory will always be a ceiling.

Self-hosted runners on AWS (originally on the table) remain a non-starter due to maintenance burden and availability risk.
EOF
)