Skip to content

Local CI pipeline: run integration-test coverage in ≤1 minute on 4-core / 16 GB (modulith-as-primary, configuration-variant scope) #4862

Open
Open
Local CI pipeline: run integration-test coverage in ≤1 minute on 4-core / 16 GB (modulith-as-primary, configuration-variant scope)#4862
Assignees
balhar-jakub
Labels
architectureIssues that have major architectural impact on the solutionenhancementNew feature or requestsize/L

Description

@balhar-jakub
balhar-jakub
opened on Jul 30, 2026

Summary

Propose a structured approach to make integration-test coverage available in a local pipeline running in ≤1 minute on 4-core / 16 GB RAM hardware.

The proposal is modulith-as-primary in scope because that's the active development target. This scoping is narrower than a full split+modulith coverage, but the Java code under test (gateway routing, ZAAS auth, JWT validation, Eureka integration) is identical for both deployment topologies — they're run with the same excludeTags list and the same @Test classes. Most split-arch jobs in the current pipeline are functionally redundant with their modulith counterparts and can be deleted from CI as a side benefit.

What the local pipeline must cover is the configuration variants that exist in CI today — SAF provider, z/OSMF without JWT, Discovery basic auth off, unknown hostnames, Infinispan storage, and the legitimate test categories (registration, gateway routing, central registry). These each exercise different code paths.

What the CI pipeline must keep (cannot run locally in 1 minute): HA topology (multi-instance + chaos matrix), JGroups/Infinispan clustering, real mainframe providers (SAF, z/OSMF, ICSF), real OIDC provider, real Node.js/Python sample apps, and real container chaos scripts.

Background

Service modules (already fast, already passing locally)

Module Test files Test methods Spring slices
gateway-service 89 486 7 @SpringBootTest
discovery-service 22 117 5
api-catalog-services 37 169 6
caching-service 33 308 6
zaas-service 72 548 8
zaas-client 7 80 —
apiml-security-common 45 240 —
apiml-common 26 103 —
apiml-tomcat-common 11 60 —
common-service-core 50 256 —
certificate-common 1 17 —
onboarding-enabler-java 7 49 —
TOTAL unit tests 400 files 2,433 methods

These run as part of ./gradlew clean build. They use Mockito / pure unit tests / @WebMvcTest slices and do not require the gateway stack.

integration-tests/ module — the slow part

196 test files, ~406 test methods. Each implements TestWithStartedInstances, which calls FullApiMediationLayer.startServices() to spin up Gateway + Discovery + ZAAS + API Catalog + Caching + Mock Services + Discoverable Client via ProcessBuilder (local) or Docker (CI).

Current CI workflow — 29 jobs

integration-tests.yml runs jobs on GH-hosted runners with 15-min timeouts. Many jobs duplicate their modulith counterparts: CITests and CITestsModulith both call runContainerTests/runContainerModulithTests with the same excludeTags list (the only excludeTags difference is NonModulithTest, which is one or two tests). Same for CITestsRegistration vs CITestsRegistrationModulith (both run runRegistrationTests), and the DiscoveryBasicAuth/ZosmfWithoutJwt pairs.

Aside: collapsing the duplicate split/modulith pairs in CI itself saves ~50% of runner minutes and is a quick win independent of this issue.

Test categories in integration-tests/ — what exercises different code paths

The Java tests themselves don't care about deployment topology. The tags that do gate different code paths are:

Configuration variant Tag Real services required Local-runnable embedded
Standard smoke (CITests, CITestsModulith) — full stack ✓ @SpringBootTest + WireMock
SAF auth provider (SAFAuthTest, SAFProviderTest) @SAFAuthTest, @SAFProviderTest Real SAF/ICSF ✗ mainframe
z/OSMF without JWT (ZosmfAuthTest) @zOSMFAuthTest real z/OSMF in ltpa mode ✗ mainframe
Unknown hostnames @UnknownHostnamesTest unresolvable hostnames ✓ embedded with bad hostname config
Discovery basic auth (DiscoveryBasicAuthTest) @DiscoveryBasicAuthTest cert-verification-off config ✓ embedded
Registration (RegistrationTest) @RegistrationTest, @MultipleRegistrationsTest full stack ✓ embedded
Caching service (CachingServiceTest, RedisReplica, RedisSentinel) @CachingServiceTest, @CachingServiceTest various storage backends In-memory: ✓; Redis: ✗
Central registry (CentralRegistryModulith) @GatewayCentralRegistry 2-gateway topology partial (5 of 7 embedded; some need 2 contexts)
Infinispan storage @InfinispanStorageTest JGroups cluster ✗ requires 2+ caching instances
HA chaotic (HA × matrix) @HATest, @ChaoticHATest multi-process + chaos scripts ✗ Docker chaos only
Sticky/deterministic LB @StickySessionLbHaTest, @DeterministicLbHaTest multi-instance gateway ✗ Docker HA only
OIDC (OidcOauth2Test, ~30 tests) @OidcOauth2Test real OIDC provider partial — about 10 of 30 can be embedded with mock provider
z/OS provider (SafLoginTest, ZosmfLoginTest, ~12 tests) @GeneralAuthenticationTest real mainframe ✗ mainframe
Node.js / Python sample apps @NodeEnablerTest, @PythonEnablerTest real Node.js / Python runtimes ✗
Penetration (JwtPenTest) — adversarial envs ✗ complex state
WebSocket / STOMP / SSE proxy (WebSocketProxyTest, StompProxyTest, ServerSentEventsProxyTest) @WebsocketTest real broker + handshake partial (2-3 of 9 WS tests embedded; STOMP/SSE: ✗)
Service ID prefix replacer (ServicePrefixReplacerIntegrationTest) — full stack ✓ embedded
Passticket / ZAAS schemes (PassticketSchemeTest, ZoweJwtSchemeTest, etc.) @GatewayServiceRouting full stack ✓ embedded with WireMock
ZAAS error paths (ZaasNegativeTest) — ZAAS only ✓ pure embedded
Personal Access Token (AccessTokenServiceTest, PATWithAllSchemesTest) — full stack partial — provider-agnostic parts embedded; provider-specific: ✗
Functional Gateway (GatewayRoutingTest, VersionTest, PageRedirectionTest, GatewayAuthenticationTest, InMemoryRateLimiterIntegrationTest, StaticClientRoutingEndpointsTest) various full stack ✓ all embedded
API Catalog (ApiCatalogEndpointIntegrationTest, ApiCatalogLoginIntegrationTest, ApiCatalogAuthenticationTest) @CatalogTest full stack ✓ all embedded
Discovery service (DiscoveryServiceAuthenticationTest, DiscoveryServiceRegistrationTest, DiscoveryBasicAuthProtectionTest) — full stack ✓ all embedded
Caching service (CachingAuthenticationTest, CachingStorageTest) — full stack + backends In-memory variant: ✓; Redis/VSAM: ✗

Why This Issue

Today, the dev loop is:

  1. Push PR
  2. Wait ~10-15 minutes for integration-tests.yml to finish
  3. Find out a 5-line change broke gateway routing

A fast local pipeline would catch most regressions before pushing. The 4-core / 16 GB constraint is what a developer's laptop has, and what a small CI runner has.

Proposed Solution

Goal

  • ./gradlew localFast runs ≤60 seconds on 4-core / 16 GB hardware
  • Catches ~85% of CI regressions locally (excluding HA/chaos/mainframe-specific)
  • Decreases CI cost by deleting redundant split/modulith duplicate jobs (independent win)

Pattern (illustrative for gateway-service)

@SpringBootTest(
    webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
    properties = {
        "apiml.enabled=false",
        "spring.profiles.active=embedded"
    },
    classes = GatewayServiceApplication.class
)
@Tag("GatewayServiceRouting")
class GatewayRoutingEmbeddedTest {
    @LocalServerPort int port;

    @RegisterExtension
    static WireMockExtension discoverableClientMock = WireMockExtension.newInstance()
        .options(wireMockConfig().dynamicPort())
        .build();

    @DynamicPropertySource
    static void register(DynamicPropertyRegistry r) {
        r.add("apiml.service.discoveryServiceUrls", () ->
            "http://localhost:" + discoverableClientMock.getPort() + "/eureka");
    }

    @ParameterizedTest
    @CsvSource({
        "/apiml1" + DISCOVERABLE_GREET,
        DISCOVERABLE_GREET,
    })
    void testRoutingWithBasePath(String basePath) {
        given().relaxedHTTPSValidation()
            .get("https://localhost:" + port + basePath)
            .then().statusCode(200);
    }
}

apiml.enabled=false disables Eureka peer replication (the source of the BuildAndTest 35-min timeout). WireMock stubs /eureka/apps and the discoverable-client routes.

Phase 1 — Configuration (2 dev-days)

  • Apply apiml.enabled=false and parallelTests plugin to all service-module test tasks
  • Add application-embedded.yml profile to each service
  • Add WireMock test dependency
  • Add @Tag("CIOnly") to integration tests that must stay CI-only (HA/chaos/mainframe)
// per-service build.gradle
test {
    forkEvery 50
    maxParallelForks 2
    useJUnitPlatform {
        if (!project.hasProperty('runSlowTests')) {
            excludeTags 'SlowTests', 'HATest', 'ChaoticHATest',
                        'InfinispanJGroupStabilityTest', 'CachingServiceTest',
                        'InfinispanStorageTest', 'RedisTest'
        }
    }
}

Phase 2 — Move tests into service modules (5 dev-weeks)

Per-service move plan with effort estimates. All counts are modulith-as-primary; split-arch equivalents of these tests are deleted from CI as a side effect.

Service Tests moved Effort Net CI jobs removed
gateway-service ~46 (gateway routing, X-509 scheme, Passticket scheme, JWT scheme, CORS, page redirect, rate limit, central registry, X-Forward, version, static client) 1.5 weeks delete GatewayProxy split, GatewayCentralRegistry split, CITestsServicePrefixReplacer split
discovery-service ~8 (auth, basic auth protection, multi-registration) 3-4 days delete CITestsDiscoveryBasicAuth split
api-catalog-services ~18 (endpoints, login, auth variants) 1 week (none — modulith has these)
caching-service ~16 (auth, in-memory storage; defer Redis/VSAM) 3-4 days delete CITestsWithRedisReplica, CITestsWithRedisSentinel (or fold into HA chaos)
zaas-service ~28 (negative, PAT, generic login, OIDC core subset) 2 weeks delete CITestsZaas split
zaas-client ~14 (E2E client library with embedded ZAAS) 3 days —
onboarding-enabler-java ~3 (token validation) 1 day —
Subtotal ~133 tests ~6.5 weeks

For each move:

  1. Pull test logic into the new module
  2. Replace TestWithStartedInstances with @SpringBootTest(webEnvironment = RANDOM_PORT) + WireMock
  3. Replace getUriFromGateway() with localhost:${localServerPort}
  4. Old test file gets @Disabled("Moved to <target-module>/.../...")
  5. After 2-4 weeks CI proves both versions agree, delete the CI counterpart

Phase 3 — Tier 2 multi-service contracts in single JVM (1.5 dev-weeks)

For flows that genuinely need gateway + z/OS auth + mock service working together, add new embedded tests that boot 2-3 services in one JVM via SpringApplication.run() with separate ports. Coverage targets where existing integration tests are too coarse:

  • MultiServiceJwkRotationTest — JWKS endpoint in ZAAS, key rotation, gateway validates
  • CrossServiceOauthFlowTest — gateway → ZAAS → back to gateway with OAuth2
  • ApiCatalogFullChainTest — Catalog → discovery → gateway → mock service
  • StaticRefreshEndToEndTest — POST /static-api/refresh → catalog → discovery reload

~6 new tests, all <300ms each once contexts are loaded.

Phase 4 — Local pipeline + CI sync (1 dev-week)

# .github/workflows/local-fast.yml
name: Local Fast (1-min target)
on: [push, pull_request]
jobs:
  fast:
    runs-on: ubuntu-latest
    timeout-minutes: 3
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: {java-version: 17, distribution: semeru}
      - run: ./gradlew test :integration-tests:tier2Suite --max-parallel-fork=2

Plus a localFast task that aggregates module-level tests with apiml.enabled=false.

Test Inventory After Migration

Source today Local runs CI runs
Standard smoke (~120 tests via runContainerModulithTests) ✓ Embedded ✓ kept for parity
SAF provider tests (~10) ✗ ✓
ZOSMF without JWT (~5) ✗ ✓
Unknown hostnames (~3) ✓ ✓
DiscoveryBasicAuth (~3) ✓ ✓
Registration (~3) ✓ ✓
CentralRegistry (~7) 5 of 7 all 7
Caching service (in-memory part) (~10) ✓ ✓
Caching service (Redis part) (~5) ✗ ✓
Infinispan (~3) ✗ ✓ needs JGroups
HA + chaos matrix (~5 × matrix) ✗ ✓
LB HA — sticky, deterministic (~10) ✗ ✓
OIDC provider (~30) ~10 with mock all 30
ZAAS mainframe auth (~12) ✗ ✓
Node/Python sample apps (~5) ✗ ✓
Penetration (~variadic) ✗ ✓
WebSocket proxy (~9) 2-3 all 9
Service ID prefix replacer (~2) ✓ ✓
Passticket / ZoweJwt schemes (~17) ✓ ✓
ZAAS error paths (~6) ✓ ✓
PAT provider-agnostic (~9) partial ✓
Functional Gateway (~15) ✓ ✓
API Catalog (~18) ✓ ✓
Discovery service (~8) ✓ ✓
Multiple-Registrations (~1) ✓ ✓
Net result ~150 tests ~330 tests

CI Job Reduction (Independent Win)

The 22-job matrix collapses. Suggested new workflow:

New job What Old equivalent
ModulithFast Standard smoke, embedded configs CITestsModulith + many
ModulithAuthProviders SAF + ZOSMF, real mainframe CITestsModulithSAFProviderHA, CITestsZosmfWithoutJwtModulith
ModulithHAChaotic HA + chaos matrix CITestsModulithHA × matrix
ModulithInfinispan JGroups mesh CITestsModulithWithInfinispan
ModulithSpecialty UnknownHostnames, CentralRegistry, BasicAuthOff, NodePython multiple modulith jobs
(deleted) — all split-arch jobs (functionally redundant)

Net: 5 jobs instead of 22, each with the same coverage but ~80% lower runner-minutes.

Acceptance Criteria

  • ./gradlew localFast runs locally on 4-core / 16 GB in ≤60 seconds
  • At least 133 currently-in-integration-tests/ tests are also executed by localFast, hosted in their respective service modules
  • All @Disabled-marked moved tests remain runnable in CI for 2-4 weeks before removal to confirm correctness
  • New pipeline file .github/workflows/local-fast.yml has 3-minute timeout
  • Local pipeline catches regressions in: gateway routing, CORS handling, ZAAS error paths, PAT lifecycle, API Catalog endpoints, cross-service JWT, discovery basic-auth configs — within 60 seconds of a developer's git push
  • As a side benefit, CI workflow job count drops from 29 to ≤8 by deleting functionally redundant split-arch duplicates

What Stays CI-Only

Cannot run locally in 1 minute due to fundamental multi-process / external-system constraints:

Category Why
authentication/providers/Saf*, Zosmf* Real mainframe required
ha/* (chaos, restart, replication) Multi-process / Docker / chaos scripts
zos/* Real mainframe
penetration/JwtPenTest Adversarial envs
discoverable-client/*Node*, *Python* Real Node.js / Python runtimes
proxy/StompProxyTest, ServerSentEventsProxyTest Real brokers / streams
proxy/WebSocketProxyTest (most) Real WS handshake timing
Redis/VSAM storage variants Real backends
OIDC tests against real provider (~20 of 30) Real OIDC
HA chaos matrix Docker multi-instance + chaos scripts
Infinispan JGroup stability Multi-node JGroups mesh

Memory & CPU Budget on 4-core / 16 GB

Process Heap
Gradle daemon 256 MB
4 forked test JVMs (1.5 GB each) 6.0 GB
OS + tooling 1.0 GB
Total ~7.25 GB / 16 GB ✓

CPU: most tests are I/O-bound (waiting for localhost HTTP response), so even single core works. --max-parallel-fork=2 avoids thrash.

Risks & Open Questions

  1. WireMock stubbing has bugs not caught by CI. Mitigation: keep both versions for 2-4 weeks before deleting CI counterparts.
  2. Some tests fundamentally cannot embed (multi-instance HA, real mainframe). Mitigation: documented in this issue, kept CI-only.
  3. Spring context caching not always effective with @DirtiesContext. Mitigation: use forkEvery, max-parallel-fork=2 to recycle JVMs.
  4. Local pipeline ≠ production behavior. Some modules (like the keyring:// SAF path) can only be exercised on z/OS. Mitigation: accept that local catches a subset, CI catches the rest.

Effort Estimate

Phase Effort
1 — Configuration 2 dev-days
2 — Move tests per service 6.5 dev-weeks (incl. parity verification, then CI cleanup)
3 — Tier 2 multi-service 1.5 dev-weeks
4 — Local pipeline + CI consolidation 1 dev-week
Total ~8 dev-weeks

Labels

  • enhancement
  • architecture
  • size/L

References

Metadata

Metadata

Assignees

Labels

architectureIssues that have major architectural impact on the solutionenhancementNew feature or requestsize/L

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions