Bound concurrent password-hash verifications and cache repeated failed credential checks - #6393
Conversation
…d credential checks The internal authentication backend runs a full password-hash verification (BCrypt) for every credential check, on the request-handling thread. Under high authentication volume this can consume significant CPU and add latency to concurrent requests. This change adds bounded concurrency and caching to the internal auth path: 1. A configurable limit on concurrent hash verifications (semaphore, default max(1, availableProcessors/4)). When the limit is reached the request receives 503 SERVICE_UNAVAILABLE via a new AuthBackendThrottledException. These responses are intentionally not reported to auth_failure_listeners, so transient load does not affect the IP/username rate limiters. 2. A short-lived cache of failed credential checks so repeated identical failures are answered without re-running the hash verification. Only definitive credential failures populate it, signalled by a new typed InvalidCredentialsException; transient errors during startup/reload continue to throw the generic OpenSearchSecurityException and are not cached. 3. An optional AuthenticationBackend#userExists() fast-path to short-circuit requests for unknown users before hash verification runs. New optional settings, all with safe defaults: plugins.security.auth.max_concurrent_bcrypt plugins.security.cache.incorrect_credential_ttl_minutes plugins.security.cache.incorrect_credential_max_size Adds unit tests for the new exception types and userExists(). Signed-off-by: Pranav Garg <garprana@amazon.com>
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 8d1f01c. ⛔ Hard block: Issues at High severity or above will block this PR from merging.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Explore these optional code suggestions:
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6393 +/- ##
==========================================
+ Coverage 75.33% 75.36% +0.02%
==========================================
Files 456 459 +3
Lines 30075 30222 +147
Branches 4564 4590 +26
==========================================
+ Hits 22657 22776 +119
- Misses 5297 5314 +17
- Partials 2121 2132 +11
🚀 New features to boost your workflow:
|
| // throttle exception propagates to the caller for 503 handling | ||
| // and security exceptions follow the original null-return path. | ||
| if (e.getCause() instanceof AuthBackendThrottledException) { | ||
| throw (AuthBackendThrottledException) e.getCause(); |
There was a problem hiding this comment.
method description say's : no auditlog, throw no exception, does also authz for all authorizers . but we are now throwing exception's.
| } | ||
| final Semaphore semaphore = bcryptSemaphore; | ||
| final boolean shouldAcquire = (semaphore != null && authBackend instanceof InternalAuthenticationBackend); | ||
| if (shouldAcquire && !semaphore.tryAcquire()) { | ||
| log.warn("BCrypt concurrency limit reached, rejecting auth for user {}", ac.getUsername()); | ||
| throw new AuthBackendThrottledException("BCrypt concurrency limit reached for user " + ac.getUsername()); | ||
| } | ||
| try { | ||
| // Narrow catch: only wrap authenticate(). authz() exceptions | ||
| // (e.g. "role not found") must NOT poison the incorrect-credential cache. | ||
| final User authenticatedUser; | ||
| try { | ||
| authenticatedUser = authBackend.authenticate(context); | ||
| } catch (InvalidCredentialsException e) { | ||
| // Definitive credential failure (wrong user / wrong password / | ||
| // empty password) thrown only by InternalAuthenticationBackend. | ||
| // Transient errors (e.g. "not configured" during startup/reload) | ||
| // keep throwing plain OpenSearchSecurityException and are NOT cached. | ||
| incorrectCredentialCache.put(ac, Boolean.TRUE); | ||
| throw e; | ||
| } |
There was a problem hiding this comment.
this whole logic is scoped to Basic Auth alone, can we have this logic inside of InternalAuthenticationBackend.authenticate() method ?
| if (authBackend instanceof InternalAuthenticationBackend && incorrectCredentialCache.getIfPresent(ac) != null) { | ||
| if (log.isDebugEnabled()) { | ||
| log.debug("Credentials for user {} found in incorrect-credential cache, rejecting", ac.getUsername()); | ||
| } | ||
| return null; | ||
| } |
There was a problem hiding this comment.
Can this lead to timing attacks ?
| remoteAddress | ||
| ); | ||
| request.queueForSending( | ||
| new SecurityResponse(SC_SERVICE_UNAVAILABLE, "Authentication service temporarily unavailable, please retry later") |
There was a problem hiding this comment.
Instead of 503, we should return 429 to the caller (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429)
| /* | ||
| Handle anonymous auth. | ||
| Populate thread context with anonymous user is anonymous login requested and no other credentials provided. | ||
| */ |
There was a problem hiding this comment.
move this comment to this if block : if (authCredentials == null && anonymousAuthEnabled && isRequestForAnonymousLogin(request.params(), request.getHeaders())) {
|
@pgtgrly
|
|
One interesting thing is that the speed of response of a system could reveal to an attacker of the existence of a valid username, given that retries with the cache will respond faster. |
We've actually previously created an advisory on that: GHSA-c6wg-cm5x-rqvj We should have some test cases around this so let's see if they give assurance or need to be updated. |
|
@pgtgrly Can you please fix the conflicts? Apologies for not being able to review this in the past 2 weeks. |
Description
Adds bounded concurrency and caching around password-hash verification in the internal authentication backend.
Category: Enhancement
Why these changes are required?
The internal authentication backend performs a full password-hash verification (BCrypt) for every credential check, on the request-handling thread. BCrypt is intentionally expensive, so under high authentication volume these verifications can consume a large share of CPU and add latency to concurrent requests. There is currently no bound on how many run at once, and repeated identical failing credentials re-run the full verification every time.
What is the old behavior before changes and new behavior after changes?
Old behavior: every credential check runs a full hash verification with unbounded concurrency; repeated identical failures each pay the full cost.
New behavior: three additions, all opt-out-able via settings and with conservative defaults:
Bounded concurrent hash verifications — a semaphore limits how many verifications run at once (default
max(1, availableProcessors / 4)). When the limit is reached, the request receives503 SERVICE_UNAVAILABLE, signalled internally by a newAuthBackendThrottledException. These responses are deliberately not reported toauth_failure_listeners, so transient load does not affect the IP/username rate limiters. Only the internal backend is limited; LDAP, SAML, Kerberos, JWT and PKI paths are untouched.Short-lived cache of failed credential checks — repeated identical failing credentials are answered from cache without re-running the verification. Only definitive credential failures populate it, signalled by a new typed
InvalidCredentialsException. Transient errors (e.g. "backend not configured" during startup or a config reload) continue to throw the genericOpenSearchSecurityExceptionand are never cached. The cache is cleared alongside the other auth caches on config reload.Optional
AuthenticationBackend#userExists()fast-path — a default method (returnsOptional.empty(), so existing custom backends are unaffected) letting a backend report whether a user is known, so requests for unknown users can be short-circuited before the hash verification runs.New optional settings, all with safe defaults:
plugins.security.auth.max_concurrent_bcryptmax(1, availableProcessors / 4)(0disables the limit)plugins.security.cache.incorrect_credential_ttl_minutes10plugins.security.cache.incorrect_credential_max_size10000Issues Resolved
None — this is a standalone improvement to the authentication path and is not tied to an existing issue.
Is this a backport? No. Requesting a
backport 3.7label so this also lands on the 3.7 branch.Does this introduce new permission(s) to be displayed in the static dropdown on the front-end? No.
Testing
Unit tests (added, all passing):
AuthBackendThrottledExceptionTest(5 tests) — message preservation, that it is an uncheckedRuntimeException, that the stack trace is suppressed (fillInStackTracereturnsthis, matching Netty'sStacklessClosedChannelExceptionpattern), and — importantly — that it does not extendOpenSearchSecurityException, so it is not absorbed by the generic auth-failure handling and can propagate to the 503 handler.InvalidCredentialsExceptionTest(3 tests) — message preservation, that it remains assignable toOpenSearchSecurityExceptionso existing callers are unchanged, and that it retains a stack trace for diagnostics.InternalAuthBackendTests(+3 tests, 7 total) —userExists()returningtrue/false, and returningOptional.empty()when theInternalUsersModelis transientlynullduring startup/reload (must not throw).Manual testing: built the plugin against 3.7.0 and ran it in a single-node Docker cluster with the limit forced to 1 permit. Verified that: existing users authenticate normally in steady state; concurrent cold-cache logins receive
503rather than401when the limit is reached; and those503s do not increment the IP rate limiter (a client on the same source IP continued to authenticate successfully throughout).Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.