ZOWE CPU consumption

[GilPen]

We notice Zowe needs more and more CPU during some 5 minutes consecutives intervals about 0,10s every 5 minutes. Just below the trace we exported from AMI OPS.

Do you confirm it's a normal behavior ? Does it mean ZOWE need CPU to keep in touch with all the BPXAS ZOWE is running ?

ZWESL1U 06JUL2026 17:05 S0006517 S STCWAS ZOWE NO 0.09 0.03 1.3 231 88.9 11.1 00:10:50.53 644:31:0 7459 Active 00:10:50

ZWESL1U 06JUL2026 17:10 S0006517 S STCWAS ZOWE NO 0.09 0.03 233 88.9 11.1 00:10:50.63 644:36:0 7309 Active 00:10:50

ZWESL1U 06JUL2026 17:15 S0006517 S STCWAS ZOWE NO 0.10 0.03 232 90.0 10.0 00:10:50.73 644:41:0 7582 Active 00:10:50

ZWESL1U 06JUL2026 17:20 S0006517 S STCWAS ZOWE NO 0.10 0.03 233 90.0 10.0 00:10:50.83 644:46:0 7526 Active 00:10:50

ZWESL1U 06JUL2026 17:25 S0006517 S STCWAS ZOWE NO 0.10 0.03 231 90.0 10.0 00:10:50.94 644:51:0 7801 Active 00:10:50

ZWESL1U 06JUL2026 17:30 S0006517 S STCWAS ZOWE NO 0.10 0.03 234 90.0 10.0 00:10:51.04 644:56:0 8047 Active 00:10:51

ZWESL1U 06JUL2026 17:35 S0006517 S STCWAS ZOWE NO 0.11 0.04 237 90.9 9.1 00:10:51.15 645:01:0 7932 Active 00:10:51

ZWESL1U 06JUL2026 17:40 S0006517 S STCWAS ZOWE NO 0.10 0.03 235 90.0 10.0 00:10:51.26 645:06:0 8229 Active 00:10:51

ZWESL1U 06JUL2026 17:45 S0006517 S STCWAS ZOWE NO 0.09 0.03 236 88.9 11.1 00:10:51.35 645:11:0 7443 Active 00:10:51

ZWESL1U 06JUL2026 17:50 S0006517 S STCWAS ZOWE NO 0.09 0.03 241 88.9 11.1 00:10:51.45 645:16:0 7198 Active 00:10:51

ZWESL1U 06JUL2026 17:55 S0006517 S STCWAS ZOWE NO 0.09 0.03 242 88.9 11.1 00:10:51.54 645:21:0 7477 Active 00:10:51

ZWESL1U 06JUL2026 18:00 S0006517 S STCWAS ZOWE NO 0.10 0.03 238 90.0 10.0 00:10:51.64 645:26:0 7732 Active 00:10:51

ZWESL1U 06JUL2026 18:05 S0006517 S STCWAS ZOWE NO 0.10 0.03 240 90.0 10.0 00:10:51.74 645:31:0 6883 Active 00:10:51

ZWESL1U 06JUL2026 18:10 S0006517 S STCWAS ZOWE NO 0.09 0.03 237 88.9 11.1 00:10:51.84 645:36:0 7707 Active 00:10:51

ZWESL1U 06JUL2026 18:15 S0006517 S STCWAS ZOWE NO 0.09 0.03 234 88.9 11.1 00:10:51.93 645:41:0 7687 Active 00:10:51

ZWESL1U 06JUL2026 18:20 S0006517 S STCWAS ZOWE NO 0.10 0.03 240 90.0 10.0 00:10:52.04 645:46:0 7684 Active 00:10:52

ZWESL1U 06JUL2026 18:25 S0006517 S STCWAS ZOWE NO 0.10 0.03 237 90.0 10.0 00:10:52.14 645:51:0 7555 Active 00:10:52

ZWESL1U 06JUL2026 18:30 S0006517 S STCWAS ZOWE NO 0.09 0.03 225 88.9 11.1 00:10:52.23 645:56:0 7534 Active 00:10:52

ZWESL1U 06JUL2026 18:35 S0006517 S STCWAS ZOWE NO 0.10 0.03 2.6 238 90.0 10.0 00:10:52.33 646:01:0 7311 Active 00:10:52

ZWESL1U 06JUL2026 18:40 S0006517 S STCWAS ZOWE NO 0.09 0.03 233 88.9 11.1 00:10:52.42 646:06:0 6896 Active 00:10:52

ZWESL1U 06JUL2026 18:45 S0006517 S STCWAS ZOWE NO 0.09 0.03 235 88.9 11.1 00:10:52.52 646:11:0 7212 Active 00:10:52

ZWESL1U 06JUL2026 18:50 S0006517 S STCWAS ZOWE NO 0.09 0.03 238 88.9 11.1 00:10:52.61 646:16:0 7197 Active 00:10:52

ZWESL1U 06JUL2026 18:55 S0006517 S STCWAS ZOWE NO 0.09 0.03 238 88.9 11.1 00:10:52.71 646:21:0 7743 Active 00:10:52

ZWESL1U 06JUL2026 19:00 S0006517 S STCWAS ZOWE NO 0.09 0.03 236 88.9 11.1 00:10:52.80 646:26:0 6849 Active 00:10:52

ZWESL1U 06JUL2026 19:05 S0006517 S STCWAS ZOWE NO 0.10 0.03 236 90.0 10.0 00:10:52.90 646:31:0 7195 Active 00:10:52

ZWESL1U 06JUL2026 19:10 S0006517 S STCWAS ZOWE NO 0.10 0.03 233 90.0 10.0 00:10:53.00 646:36:0 7796 Active 00:10:53

ZWESL1U 06JUL2026 19:15 S0006517 S STCWAS ZOWE NO 0.10 0.03 236 90.0 10.0 00:10:53.10 646:41:0 8125 Active 00:10:53

Regards

Gilles

[balhar-jakub]

Hi GilPen,

For the multi-service deployment the pattern of the approx. 0.1 usage is typical and comes from the heartbeat between the services.

Have you tried switching to single service deployment? This overhead shoul be reduced significantly there.

[GilPen]

Hello Jakub

The zowe.yaml is already switched to single service deployment.

We generated new reports.

We can see that the 2.17 version behavior is correct. The CPU stopped to increase when the activity slowed and the lift occurs after the 3.4 migration. ZOWE needs always more and more CPU and never released until sopping ZOWE.

In the second report, we can see the CPU stopped to increase when we stopped ZOWE June 4th 9h00.

Gilles Peniguel

[balhar-jakub]

Thank you, I am looking into it, I have a few potential issues that may be causing this, I need to try to replicate which may take me some time, though.

[balhar-jakub]

I continue looking into it. So far I managed to explore two potential causes with negative results:

• JWT Caching not cleansed properly
• LoadBalancerCache not cleansed properly.

[balhar-jakub]

I continue trying to replicate, these are the hypothesis and tests that I tried in the last few days, so far none of them reproduced the behavior. I have few more ideas on what to test that I continue working on.

#	Hypothesis	How it was tested	Result
1	Infinispan distributed caches (invalidatedJwtTokens, zoweCache, etc.) grow unboundedly — no maxCount/TTL	Config unit tests (would fail today) + load test: 30K/123K unique repeated login/logout cycles under SAF auth, tracking JVM heap	Plateaued after warm-up — ruled out under SAF/PassTicket. Caveat: SAF may not populate this cache the same way zosmf/JWT auth would; not retested under zosmf.
2	InMemoryRateLimiter cache (ConcurrentHashMap, no eviction) grows with unique route/service IDs	Load test with up to 123,000 unique IDs, tracking JVM heap	Small run showed a weak positive blip; scaled retest (123K) was flat — ruled out (blip was noise).
3	LoadBalancerCache orphaned entries never cleaned up	Same combined load test as #2, unique users	Flat — ruled out, same caveat as above (single-JVM consolidated mode, no remote blind spot).
4	Native/OMVS thread growth, not JVM-visible (explains 231→242 AMI OPS trace)	12h soak measuring os_thread_count via ps (not actuator) on traffic + idle-control instance	Flat for the full 12h on both — ruled out, first time measured directly rather than inferred.
5	Eureka re-registration/lease-renewal task accumulation under peer churn	20 stop/start cycles of a same-LPAR peer instance, watching /actuator/scheduledtasks	Exactly 3 tasks, zero variance across 20 cycles — ruled out for this churn pattern. Reconfirmed in the later 12h soak.
6	Metrics/Micrometer cardinality growth from per-route/per-client tags	12h soak sending 3,631 requests (30%) with deliberately unique paths, tracking tag cardinality	Flat — ruled out. Spring's URI-template normalization protects the metric as suspected.
7	Netty/Reactor connection-pool or event-loop leak on southbound retries	12h soak, tracking request latency trend over time	Latency flat, no drift — weak negative, but inconclusive: the soak's "success" traffic never got real 200s (auth gap), so the retry/circuit-breaker path wasn't truly exercised.
8	H-A: periodicJwtInit stuck-zOSMF busy loop (3.x/Modulith-only, retries every 20s forever if zOSMF never ready)	Forced zOSMF to connection refused, idled ~72 min under provider: zosmf, watched logs + CPU	Does not reproduce as hypothesized — ruled out for clean connection-refused; retries stopped after 3 attempts, CPU near-zero. Caveat: untested for a slow/hanging zOSMF instead of an immediate refusal.

[GilPen]

Hello Jakub

Just for your information, in zowe.yaml the caching service is disable because we don't use hi-avaiability behavior. Do you think it can decrease CPU consumption if I activate caching ?

zowe.yaml
...
caching-service:
enabled: false
...

Regards
Gilles

[balhar-jakub]

No, I don't think it will decrease CPU consumption, but I will keep this in mind when trying to continue replication, that you do not run in HA.

[balhar-jakub]

Does your setup use AT-TLS for Network setup?