fix: escalate peer read timeouts after max retries — GH#4777

[balhar-jakub]

Closes #4777

Problem

The peer-node-read-timeout-ms was too short (15s), causing infinite retry loops under load on z/OS. Read timeouts were never escalated to permanent errors.

Fix

• Added networkIssueCounter.hasReachedMax() gate to read timeout handling in ApimlPeerEurekaNode
• After maxPeerRetries consecutive read timeout failures → PermanentError (matching connection-failure behavior)
• Increased default peer-node-read-timeout-ms from 15000 → 30000ms
• Reset counter on successful replication
• 4 new parametrized tests covering escalation, reset, and unchanged behavior

Files changed

• apiml-common/.../ApimlPeerEurekaNode.java: hasReachedMax() gate in both process(task) and process(tasks)
• discovery-service/.../application.yml: timeout 15000 → 30000
• apiml/.../application.yml: timeout 15000 → 30000
• apiml-common/.../ReplicationTaskProcessorTest.java: 4 new tests (19 total)

Build: PASS (19/19 tests)

[balhar-jakub]

QA + Security Review (Pavel's Lens) — APPROVED ✓

Pavel's Lens Results

Rule	Check	Verdict
Rule 1: Config Consistency	Both application.yml files updated identically (15000→30000)	✓ PASS
Rule 2: Deduplication	hasReachedMax() gate in both process(task) and process(tasks) — minimal, necessary duplication	✓ PASS
Rule 3: Null Safety	networkIssueCounter is initialized field; getCountText() returns StringBuilder.toString() (never null)	✓ PASS
Rule 4: Test Parametrization	4 standalone tests covering escalation, reset, regression	✓ PASS
Rule 5: Security Boundaries	No auth/authz/TLS changes; error messages don't expose sensitive data	✓ PASS
Rule 6: z/OS Awareness	Timeout increase (30s) specifically for z/OS batch replication latency	✓ PASS
Rule 7: Log Quality	Clear log message with retry count; existing warning preserved	✓ PASS
Rule 8: TODO Tracking	No new TODOs introduced	✓ PASS

Acceptance Criteria Verification

• AC1 ✓: hasReachedMax() gate escalates to PermanentError after N retries in both methods — covered by tests
• AC2 ✓: Successful replication resets counter — covered by whenSuccessfulReplicationResetsReadTimeoutCounter test
• AC3 ✓: Connection failure behavior unchanged — covered by whenConnectionFailureRepeatedMultipleTimes_thenBehaviorUnchanged test
• AC4 ✓: 4 tests covering escalation, reset, and unchanged behavior (19/19 pass)
• AC5 ✓: Default timeout increased to 30000ms in both config files

Security Review

• No sensitive data in error messages (only generic "read timed out" and retry count)
• No new endpoints or auth changes
• No secrets/credentials exposed
• PermanentError drops task; peer catches up via registry sync — no DoS vector

Verdict: APPROVED — Ready for merge. All acceptance criteria met, all 8 Pavel rules passed, no security concerns.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[balhar-jakub]

Multi-Reviewer Review

Verdict: Approve with minor NITs.

Findings:

• ℹ️ NIT — Duplicate hasReachedMax() checks in process(ReplicationTask) and process(List<ReplicationTask>) (lines 421 and 462). Extract a helper.
• ℹ️ NIT — Log messages don't use the project's ApimlLogger + message ID pattern. Consider adding a new ID in common-log-messages.yml (e.g. org.zowe.apiml.common.peerReplicationPermanentError).
• ℹ️ NIT — peer-node-read-timeout-ms doubled (15000 → 30000) — sensible, but add a comment explaining the link to GH#4777.
• ✅ Test coverage is solid (boundary case at DEFAULT_MAX_RETRIES, counter reset on success).

Good:

• Addresses achmelo's prior comment about the two log messages being too similar (the new text is now distinct).
• Behavioral change is well-reasoned: socket read timeouts escalate to PermanentError after maxPeerRetries instead of looping forever as Congestion. Catches the case where a peer is permanently broken.
• application.yml change is documented via the commit message (GH#4777).
• New imports (SocketTimeoutException) are minimal and scoped.

Required before merge: none. All findings are NIT-level.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud