ACL client caches stale "not found" for valid workload tokens

[paras-gupta2]

Description

On some client nodes, Consul can start repeatedly saying “ACL not found” for Connect/xDS requests even when the proxy token is still valid. When that happens, sidecar proxies lose xDS updates, log stream errors, and some workloads on that node lose mesh connectivity until someone intervenes.

Root Cause

Token lookup is first done with a stale read for speed. If that stale read hits a follower that is briefly behind replication, it can wrongly return “ACL not found” for a token that actually exists. That bad result then gets treated like a real missing token in cache-refresh behavior, so the node can stay stuck in that error state.

Resolution

When a stale token lookup returns ACL not found, Consul now performs a second, consistent (leader) read before treating the token as missing. This prevents temporary replication lag from causing valid tokens to be incorrectly marked as missing and poisoning the cache. If the token is genuinely deleted, the leader still returns ACL not found, so the existing behavior remains unchanged.

Testing

Added a unit test to verify that a stale ACL not found followed by a successful leader read correctly resolves and caches the token.
Confirmed the test reproduces the original issue when the fix is disabled.

[github-actions]

Go Test Coverage: 71.7%

See the workflow run for the full per-package breakdown and downloadable HTML report.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.93939% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.67%. Comparing base (a4925b9) to head (6d8ad16).
⚠️ Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
agent/consul/acl.go	93.93%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23732      +/-   ##
==========================================
+ Coverage   61.48%   65.67%   +4.19%     
==========================================
  Files         828      951     +123     
  Lines      105994   114946    +8952     
==========================================
+ Hits        65166    75489   +10323     
+ Misses      34561    32425    -2136     
- Partials     6267     7032     +765     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.