Skip to content

Bound concurrent password-hash verifications and cache repeated failed credential checks - #6393

Open
pgtgrly wants to merge 1 commit into
opensearch-project:mainfrom
pgtgrly:bound-concurrent-hash-verification
Open

Bound concurrent password-hash verifications and cache repeated failed credential checks#6393
pgtgrly wants to merge 1 commit into
opensearch-project:mainfrom
pgtgrly:bound-concurrent-hash-verification

Conversation

@pgtgrly

@pgtgrly pgtgrly commented Aug 13, 2026

Copy link
Copy Markdown

Description

Adds bounded concurrency and caching around password-hash verification in the internal authentication backend.

  • Category: Enhancement

  • Why these changes are required?

    The internal authentication backend performs a full password-hash verification (BCrypt) for every credential check, on the request-handling thread. BCrypt is intentionally expensive, so under high authentication volume these verifications can consume a large share of CPU and add latency to concurrent requests. There is currently no bound on how many run at once, and repeated identical failing credentials re-run the full verification every time.

  • What is the old behavior before changes and new behavior after changes?

    Old behavior: every credential check runs a full hash verification with unbounded concurrency; repeated identical failures each pay the full cost.

    New behavior: three additions, all opt-out-able via settings and with conservative defaults:

    1. Bounded concurrent hash verifications — a semaphore limits how many verifications run at once (default max(1, availableProcessors / 4)). When the limit is reached, the request receives 503 SERVICE_UNAVAILABLE, signalled internally by a new AuthBackendThrottledException. These responses are deliberately not reported to auth_failure_listeners, so transient load does not affect the IP/username rate limiters. Only the internal backend is limited; LDAP, SAML, Kerberos, JWT and PKI paths are untouched.

    2. Short-lived cache of failed credential checks — repeated identical failing credentials are answered from cache without re-running the verification. Only definitive credential failures populate it, signalled by a new typed InvalidCredentialsException. Transient errors (e.g. "backend not configured" during startup or a config reload) continue to throw the generic OpenSearchSecurityException and are never cached. The cache is cleared alongside the other auth caches on config reload.

    3. Optional AuthenticationBackend#userExists() fast-path — a default method (returns Optional.empty(), so existing custom backends are unaffected) letting a backend report whether a user is known, so requests for unknown users can be short-circuited before the hash verification runs.

    New optional settings, all with safe defaults:

    Setting Default
    plugins.security.auth.max_concurrent_bcrypt max(1, availableProcessors / 4) (0 disables the limit)
    plugins.security.cache.incorrect_credential_ttl_minutes 10
    plugins.security.cache.incorrect_credential_max_size 10000

Issues Resolved

None — this is a standalone improvement to the authentication path and is not tied to an existing issue.

Is this a backport? No. Requesting a backport 3.7 label so this also lands on the 3.7 branch.

Does this introduce new permission(s) to be displayed in the static dropdown on the front-end? No.

Testing

  • Unit tests (added, all passing):

    • AuthBackendThrottledExceptionTest (5 tests) — message preservation, that it is an unchecked RuntimeException, that the stack trace is suppressed (fillInStackTrace returns this, matching Netty's StacklessClosedChannelException pattern), and — importantly — that it does not extend OpenSearchSecurityException, so it is not absorbed by the generic auth-failure handling and can propagate to the 503 handler.
    • InvalidCredentialsExceptionTest (3 tests) — message preservation, that it remains assignable to OpenSearchSecurityException so existing callers are unchanged, and that it retains a stack trace for diagnostics.
    • InternalAuthBackendTests (+3 tests, 7 total) — userExists() returning true/false, and returning Optional.empty() when the InternalUsersModel is transiently null during startup/reload (must not throw).
  • Manual testing: built the plugin against 3.7.0 and ran it in a single-node Docker cluster with the limit forced to 1 permit. Verified that: existing users authenticate normally in steady state; concurrent cold-cache logins receive 503 rather than 401 when the limit is reached; and those 503s do not increment the IP rate limiter (a client on the same source IP continued to authenticate successfully throughout).

Check List

  • New functionality includes testing
  • New functionality has been documented
  • New Roles/Permissions have a corresponding security dashboards plugin PR
  • API changes companion pull request created
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…d credential checks

The internal authentication backend runs a full password-hash verification
(BCrypt) for every credential check, on the request-handling thread. Under
high authentication volume this can consume significant CPU and add latency
to concurrent requests.

This change adds bounded concurrency and caching to the internal auth path:

1. A configurable limit on concurrent hash verifications (semaphore, default
   max(1, availableProcessors/4)). When the limit is reached the request
   receives 503 SERVICE_UNAVAILABLE via a new AuthBackendThrottledException.
   These responses are intentionally not reported to auth_failure_listeners,
   so transient load does not affect the IP/username rate limiters.

2. A short-lived cache of failed credential checks so repeated identical
   failures are answered without re-running the hash verification. Only
   definitive credential failures populate it, signalled by a new typed
   InvalidCredentialsException; transient errors during startup/reload
   continue to throw the generic OpenSearchSecurityException and are not
   cached.

3. An optional AuthenticationBackend#userExists() fast-path to short-circuit
   requests for unknown users before hash verification runs.

New optional settings, all with safe defaults:
  plugins.security.auth.max_concurrent_bcrypt
  plugins.security.cache.incorrect_credential_ttl_minutes
  plugins.security.cache.incorrect_credential_max_size

Adds unit tests for the new exception types and userExists().

Signed-off-by: Pranav Garg <garprana@amazon.com>
@github-actions

Copy link
Copy Markdown
Contributor

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 8d1f01c.

Hard block: Issues at High severity or above will block this PR from merging.

PathLineSeverityDescription
src/main/java/org/opensearch/security/auth/BackendRegistry.java455mediumThe userExists() pre-check skips BCrypt entirely for non-existent users, breaking constant-time authentication behavior. Attackers can enumerate valid usernames through response-time differences: non-existent users return faster (no BCrypt delay) while existing users incur the full BCrypt cost. The original code implicitly protected against user enumeration by always running BCrypt regardless of user existence.
src/main/java/org/opensearch/security/auth/BackendRegistry.java129lowThe incorrectCredentialCache uses AuthCredentials as a cache key. If AuthCredentials.equals()/hashCode() is scoped to username only (rather than username+password), an attacker could poison the cache for a valid user by submitting one failed attempt, causing all subsequent login attempts for that user to be rejected for the TTL window (default 10 minutes). The correctness of this cache depends entirely on the AuthCredentials equality contract, which is not validated or documented here.

The table above displays the top 10 most important findings.

Total: 2 | Critical: 0 | High: 0 | Medium: 1 | Low: 1


Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.


⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

@github-actions

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 Multiple PR themes

Sub-PR theme: Add userExists() fast-path to AuthenticationBackend

Relevant files:

  • src/main/java/org/opensearch/security/auth/AuthenticationBackend.java
  • src/main/java/org/opensearch/security/auth/internal/InternalAuthenticationBackend.java
  • src/test/java/org/opensearch/security/auth/InternalAuthBackendTests.java

Sub-PR theme: Introduce AuthBackendThrottledException for BCrypt semaphore

Relevant files:

  • src/main/java/org/opensearch/security/auth/AuthBackendThrottledException.java
  • src/test/java/org/opensearch/security/auth/AuthBackendThrottledExceptionTest.java

Sub-PR theme: Add typed InvalidCredentialsException for definitive failures

Relevant files:

  • src/main/java/org/opensearch/security/auth/InvalidCredentialsException.java
  • src/test/java/org/opensearch/security/auth/InvalidCredentialsExceptionTest.java

⚡ Recommended focus areas for review

Duplicate failure notification

When userExists() returns false, earlyFailureNotified is set and failure listeners are notified immediately. However, authcz() is still invoked afterward (it will return null since credentials don't match). The early notification happens BEFORE authentication actually runs, and if the user is found in a later auth domain, the IP rate limiter has already been incremented for a user that legitimately authenticated elsewhere. This can cause spurious IP blocking for valid users who exist only in later-configured backends.

// Pre-increment IP rate limiter for non-existent users before BCrypt runs
boolean earlyFailureNotified = false;
if (ac != null) {
    Optional<Boolean> userExistsResult = authDomain.getBackend().userExists(ac.getUsername());
    if (userExistsResult.isPresent() && !userExistsResult.get()) {
        if (isDebugEnabled) {
            log.debug(
                "User {} does not exist in backend {}, notifying failure listeners early",
                ac.getUsername(),
                authDomain.getBackend().getType()
            );
        }
        for (AuthFailureListener authFailureListener : this.authBackendFailureListeners.get(
            authDomain.getBackend().getClass().getName()
        )) {
            authFailureListener.onAuthFailure(
                request.getRemoteAddress().map(InetSocketAddress::getAddress).orElse(null),
                ac,
                request
            );
        }
        earlyFailureNotified = true;
    }
}
Broken brace structure

The diff shows the old for loop over authBackendFailureListeners was replaced with an if (!earlyFailureNotified) { ... } block, but the closing brace structure looks off — the original code had one closing } for the for-loop, while the new code adds a wrapping if block. Verify that the resulting brace balance is correct and that the continue; still belongs to the outer authDomain loop rather than being orphaned inside the new conditional.

    if (!earlyFailureNotified) {
        for (AuthFailureListener authFailureListener : this.authBackendFailureListeners.get(
            authDomain.getBackend().getClass().getName()
        )) {
            authFailureListener.onAuthFailure(
                request.getRemoteAddress().map(InetSocketAddress::getAddress).orElse(null),
                ac,
                request
            );
        }
    }
    continue;
}
Cache poisoning risk

incorrectCredentialCache is keyed by AuthCredentials (which includes the password). An attacker sending many wrong-password attempts for a valid username can fill the cache up to max_size (default 10,000) with permutations, potentially evicting legitimate cached failures and consuming memory. Consider whether the cache key should exclude the password or if per-username throttling would be more effective.

incorrectCredentialCache = CacheBuilder.newBuilder()
    .expireAfterWrite(
        opensearchSettings.getAsInt(
            ConfigConstants.SECURITY_CACHE_INCORRECT_CREDENTIAL_TTL_MINUTES,
            ConfigConstants.SECURITY_CACHE_INCORRECT_CREDENTIAL_TTL_MINUTES_DEFAULT
        ),
        TimeUnit.MINUTES
    )
    .maximumSize(
        opensearchSettings.getAsInt(
            ConfigConstants.SECURITY_CACHE_INCORRECT_CREDENTIAL_MAX_SIZE,
            ConfigConstants.SECURITY_CACHE_INCORRECT_CREDENTIAL_MAX_SIZE_DEFAULT
        )
    )
    .build();
Semaphore leak on throttle

In the Callable, if tryAcquire() fails the code throws AuthBackendThrottledException without acquiring — correct. However, shouldAcquire is captured before the throw, and the finally block only releases when shouldAcquire is true. Since the throw happens on the tryAcquire-false branch, no permit was acquired, so no release is needed — but note that shouldAcquire remains true and could mask logic errors during refactors. Consider using a local acquired flag set only after successful acquisition to make the invariant explicit.

final Semaphore semaphore = bcryptSemaphore;
final boolean shouldAcquire = (semaphore != null && authBackend instanceof InternalAuthenticationBackend);
if (shouldAcquire && !semaphore.tryAcquire()) {
    log.warn("BCrypt concurrency limit reached, rejecting auth for user {}", ac.getUsername());
    throw new AuthBackendThrottledException("BCrypt concurrency limit reached for user " + ac.getUsername());
}
try {
    // Narrow catch: only wrap authenticate(). authz() exceptions
    // (e.g. "role not found") must NOT poison the incorrect-credential cache.
    final User authenticatedUser;
    try {
        authenticatedUser = authBackend.authenticate(context);
    } catch (InvalidCredentialsException e) {
        // Definitive credential failure (wrong user / wrong password /
        // empty password) thrown only by InternalAuthenticationBackend.
        // Transient errors (e.g. "not configured" during startup/reload)
        // keep throwing plain OpenSearchSecurityException and are NOT cached.
        incorrectCredentialCache.put(ac, Boolean.TRUE);
        throw e;
    }
    return authz(context, authenticatedUser, roleCache, authorizers);
} finally {
    if (shouldAcquire) {
        semaphore.release();
    }
}

@github-actions

Copy link
Copy Markdown
Contributor

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
Security
Avoid username-enumeration timing side channel

Calling userExists() before authenticate() and short-circuiting failure
notifications creates a timing side channel that lets attackers enumerate valid
usernames: nonexistent users trigger an immediate failure listener call and skip
BCrypt, whereas existing users incur BCrypt latency. This defeats the constant-time
anti-enumeration property that the original code preserved. Consider removing the
pre-check or applying it only after credential verification.

src/main/java/org/opensearch/security/auth/BackendRegistry.java [458-462]

-// Pre-increment IP rate limiter for non-existent users before BCrypt runs
+// NOTE: skip pre-check to preserve constant-time behavior and avoid
+// username enumeration via timing / rate-limiter side channels.
 boolean earlyFailureNotified = false;
-if (ac != null) {
-    Optional<Boolean> userExistsResult = authDomain.getBackend().userExists(ac.getUsername());
-    if (userExistsResult.isPresent() && !userExistsResult.get()) {
Suggestion importance[1-10]: 8

__

Why: This is a valid and important security concern: the pre-check userExists() introduces a timing/behavior side channel that enables username enumeration, undermining constant-time authentication properties.

Medium
Avoid caching plaintext credentials in memory

Populating incorrectCredentialCache with the raw AuthCredentials (which holds the
plaintext password bytes) keeps sensitive credentials pinned in heap memory for the
entire TTL (default 10 minutes), and also relies on AuthCredentials.equals/hashCode
— if these compute over mutable/cleared password state the cache lookup on retry may
not hit. Consider caching a derived key (e.g. username + salted hash of the
password) instead of the full credentials object, so plaintext isn't retained.

src/main/java/org/opensearch/security/auth/BackendRegistry.java [932]

 try {
     authenticatedUser = authBackend.authenticate(context);
 } catch (InvalidCredentialsException e) {
-    // Definitive credential failure (wrong user / wrong password /
-    // empty password) thrown only by InternalAuthenticationBackend.
-    // Transient errors (e.g. "not configured" during startup/reload)
-    // keep throwing plain OpenSearchSecurityException and are NOT cached.
+    // Cache a derived key rather than raw credentials to avoid
+    // pinning plaintext password bytes in memory for the TTL.
     incorrectCredentialCache.put(ac, Boolean.TRUE);
     throw e;
 }
Suggestion importance[1-10]: 6

__

Why: Valid concern about pinning plaintext password bytes in the cache for the TTL duration, but the improved_code is identical to the existing_code so it does not actually demonstrate the fix. Still, the security consideration is worth raising.

Low
Ensure cached failures still penalize IP

Returning null here bypasses the AuthFailureListener notification loop in the caller
only when earlyFailureNotified is false — but the IP rate limiter should still be
incremented on cached failure to prevent attackers from cheaply retrying known-bad
passwords without cost. Verify that the caller's post-authcz==null failure listener
path still runs (it does, since earlyFailureNotified guards it), otherwise this
becomes a rate-limit bypass.

src/main/java/org/opensearch/security/auth/BackendRegistry.java [884-889]

 if (authBackend instanceof InternalAuthenticationBackend && incorrectCredentialCache.getIfPresent(ac) != null) {
     if (log.isDebugEnabled()) {
         log.debug("Credentials for user {} found in incorrect-credential cache, rejecting", ac.getUsername());
     }
+    // fall through to caller so AuthFailureListener is invoked and IP is penalized
     return null;
 }
Suggestion importance[1-10]: 3

__

Why: The suggestion mainly asks to verify caller behavior and acknowledges the current flow likely works correctly; it does not propose a concrete change. Low impact.

Low
General
Use bounded wait instead of immediate reject

A non-blocking tryAcquire() causes immediate 503 rejection under any brief spike,
even when a permit would free up in milliseconds. Consider using tryAcquire(timeout,
unit) with a small bounded wait (e.g. configurable, defaulting to a few hundred ms)
so legitimate traffic queues briefly rather than being rejected outright, while
still bounding worst-case latency.

src/main/java/org/opensearch/security/auth/BackendRegistry.java [915-920]

 final Semaphore semaphore = bcryptSemaphore;
 final boolean shouldAcquire = (semaphore != null && authBackend instanceof InternalAuthenticationBackend);
-if (shouldAcquire && !semaphore.tryAcquire()) {
+if (shouldAcquire && !semaphore.tryAcquire(bcryptAcquireTimeoutMs, TimeUnit.MILLISECONDS)) {
     log.warn("BCrypt concurrency limit reached, rejecting auth for user {}", ac.getUsername());
     throw new AuthBackendThrottledException("BCrypt concurrency limit reached for user " + ac.getUsername());
 }
Suggestion importance[1-10]: 5

__

Why: Reasonable enhancement to smooth over brief spikes, but the current design is intentional (fast-fail with 503) and this is a debatable trade-off rather than a defect.

Low

@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.00000% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.36%. Comparing base (5e8e5f1) to head (8d1f01c).
⚠️ Report is 11 commits behind head on main.

Files with missing lines Patch % Lines
.../org/opensearch/security/auth/BackendRegistry.java 58.13% 25 Missing and 11 partials ⚠️
...y/auth/internal/InternalAuthenticationBackend.java 85.71% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #6393      +/-   ##
==========================================
+ Coverage   75.33%   75.36%   +0.02%     
==========================================
  Files         456      459       +3     
  Lines       30075    30222     +147     
  Branches     4564     4590      +26     
==========================================
+ Hits        22657    22776     +119     
- Misses       5297     5314      +17     
- Partials     2121     2132      +11     
Files with missing lines Coverage Δ
...h/security/auth/AuthBackendThrottledException.java 100.00% <100.00%> (ø)
...pensearch/security/auth/AuthenticationBackend.java 100.00% <100.00%> (ø)
...rch/security/auth/InvalidCredentialsException.java 100.00% <100.00%> (ø)
...g/opensearch/security/support/ConfigConstants.java 95.65% <100.00%> (ø)
...y/auth/internal/InternalAuthenticationBackend.java 79.41% <85.71%> (+1.28%) ⬆️
.../org/opensearch/security/auth/BackendRegistry.java 76.02% <58.13%> (-4.03%) ⬇️

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

// throttle exception propagates to the caller for 503 handling
// and security exceptions follow the original null-return path.
if (e.getCause() instanceof AuthBackendThrottledException) {
throw (AuthBackendThrottledException) e.getCause();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

method description say's : no auditlog, throw no exception, does also authz for all authorizers . but we are now throwing exception's.

Comment on lines +914 to +934
}
final Semaphore semaphore = bcryptSemaphore;
final boolean shouldAcquire = (semaphore != null && authBackend instanceof InternalAuthenticationBackend);
if (shouldAcquire && !semaphore.tryAcquire()) {
log.warn("BCrypt concurrency limit reached, rejecting auth for user {}", ac.getUsername());
throw new AuthBackendThrottledException("BCrypt concurrency limit reached for user " + ac.getUsername());
}
try {
// Narrow catch: only wrap authenticate(). authz() exceptions
// (e.g. "role not found") must NOT poison the incorrect-credential cache.
final User authenticatedUser;
try {
authenticatedUser = authBackend.authenticate(context);
} catch (InvalidCredentialsException e) {
// Definitive credential failure (wrong user / wrong password /
// empty password) thrown only by InternalAuthenticationBackend.
// Transient errors (e.g. "not configured" during startup/reload)
// keep throwing plain OpenSearchSecurityException and are NOT cached.
incorrectCredentialCache.put(ac, Boolean.TRUE);
throw e;
}

@devardee devardee Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this whole logic is scoped to Basic Auth alone, can we have this logic inside of InternalAuthenticationBackend.authenticate() method ?

Comment on lines +884 to +889
if (authBackend instanceof InternalAuthenticationBackend && incorrectCredentialCache.getIfPresent(ac) != null) {
if (log.isDebugEnabled()) {
log.debug("Credentials for user {} found in incorrect-credential cache, rejecting", ac.getUsername());
}
return null;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this lead to timing attacks ?

remoteAddress
);
request.queueForSending(
new SecurityResponse(SC_SERVICE_UNAVAILABLE, "Authentication service temporarily unavailable, please retry later")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of 503, we should return 429 to the caller (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429)

Comment on lines 604 to 607
/*
Handle anonymous auth.
Populate thread context with anonymous user is anonymous login requested and no other credentials provided.
*/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move this comment to this if block : if (authCredentials == null && anonymousAuthEnabled && isRequestForAnonymousLogin(request.params(), request.getHeaders())) {

@kkhatua

kkhatua commented Aug 14, 2026

Copy link
Copy Markdown
Member

@pgtgrly
A couple of minor comments:

  1. max(1, availableProcessors / 4) --> Why did we settle for this? Could you share some perf numbers around this?
  2. If the cache's purpose is to block recomputation of already failed/invalid creds, wouldn't an attacker essentially keep trying different permutations?
  3. Why are we holding the password in plaintext? Isnt the hash sufficient to confirm if the retry is a match? It is possible that a typo in the username and a valid password might be stored, and a heapdump would reveal this.

@kkhatua

kkhatua commented Aug 14, 2026

Copy link
Copy Markdown
Member

One interesting thing is that the speed of response of a system could reveal to an attacker of the existence of a valid username, given that retries with the cache will respond faster.
nit: This might feel like a bit of an overkill, but would it make sense to inject a delay similar to the time for a valid authentication?

@cwperks

cwperks commented Aug 18, 2026

Copy link
Copy Markdown
Member

One interesting thing is that the speed of response of a system could reveal to an attacker of the existence of a valid username, given that retries with the cache will respond faster. nit: This might feel like a bit of an overkill, but would it make sense to inject a delay similar to the time for a valid authentication?

We've actually previously created an advisory on that: GHSA-c6wg-cm5x-rqvj

We should have some test cases around this so let's see if they give assurance or need to be updated.

@cwperks

cwperks commented Aug 25, 2026

Copy link
Copy Markdown
Member

@pgtgrly Can you please fix the conflicts? Apologies for not being able to review this in the past 2 weeks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants