feat: Allow aws-vault to safely be run in parallel

[timvisher-dd]

Closes 99designs/aws-vault#1275 (abandoned)

I run aws-vault as a credentials_provider for SSO backed AWS profiles. This is often in parallel contexts of dozens to hundreds of invocations at once.

In that context when credentials needed to be refreshed, aws-vault would open an unconstrained amount of browser tabs in parallel, usually triggering HTTP 500 responses on the AWS side and failing to acquire creds.

To mitigate this I developed a small wrapper around aws-vault that would use a Bash dir lock (Wooledge BashFAQ 45) when there was a possibility that the credentials would need to be refreshed. This worked but it was also quite slow as it would lock the entire aws-vault execution rather than just the step that might launch a browser tab. The dir locking strategy was also sensitive to being killed by process managers like terraform and so had to do things like die voluntarily after magic amounts of seconds to avoid being SIGKILLed and leaving a stale lock around.

This changeset introduces a cross-platform process lock that is tolerant of all forms of process death (even SIGKILL), and wires it into the SSO/OIDC browser flow so only one process refreshes at a time. This keeps parallel invocations fast while avoiding the browser storm that triggers HTTP 500s.

While stress testing I also found that the session cache could race across processes, leading to "item already exists" errors from the keychain. This branch adds a session cache refresh lock so only one process writes back to the same cache entry at once, while others wait and re-check.

Because the parallelism is now safe and fast, I also hit SSO OIDC rate limits (429). This change adds Retry-After support with jittered backoff so the SSO token exchange is resilient under heavy load.

In a stress test across 646 AWS profiles in 9 SSO Directories I'm able to retrieve creds successfully in ~36 seconds on my box. This fails on upstream HEAD because the browser storm overwhelms the IAM/SSO APIs and the keychain cache update races.

Performance implications

This change serializes keychain operations (for the macOS keychain backend only) so only one aws-vault process is touching the keychain at a time. This avoids concurrent unlock prompts and other keychain contention at the cost of making highly parallel workloads wait their turn for keychain access. In practice this has been acceptable for my workloads.

On my machine, successfully retrieving creds for 642 profiles completes in under 60 seconds, which is good enough that I did not do a formal before/after benchmark. If maintainers want more detailed measurements I can provide them.

Testing

Unit tests have been added for the SSO/OIDC lock, session cache lock, and OIDC retry logic. I also used some local integration test scripts that clear out the creds and run aws-vault export --format=json across different sets of my profiles and assert that it succeeds. Finally I've converted my local tooling to use this fork of aws-vault and have been exercising it there without issue.

Colofon

I did not write any of this code. Codex did. That said I have read through it in some detail and it looks reasonable to me.

Co-authored-by: Codex noreply@openai.com

[mbevc1]

Thanks, sorry but we might have talked about some of the feedback, but context is a bit lost now :)

So PR still looks good and I think just a bit more feedback to polish the codebase.

• The login command is missing --parallel-safe wiring
  exec, export, and rotate all receive the flag, but login (which also opens a browser and calls SSO) is not in the diff. If a user sets AWS_VAULT_PARALLEL_SAFE=true, login will silently not benefit from the lock, which is the most visible browser-storm scenario. Either add it to login or document clearly that login is intentionally excluded and why.
• The USAGE.md documentation says the lock "applies to all backends" — but then the trade-offs section correctly notes the serialization cost. The inconsistency is in the code: the lock is applied in global.go whenever --parallel-safe is set, regardless of backend. For the file backend or pass backend, the underlying operations are likely already atomic at the OS level and the lock adds pure overhead. Consider gating the LockedKeyring wrapper on a.KeyringBackend == "keychain" (or making the behavior clearly documented per-backend), to avoid unnecessary serialization on Linux.

[mbevc1]

The fallback "aws-vault" is a fixed string shared across all instances regardless of backend. On Linux with the file or pass backend, two users on the same machine or two different aws-vault configs would contend on the same lock file path. The lock key should incorporate something backend-specific (e.g., the backend type, or the file-based keyring directory). Consider deriving the key from the full keyring config rather than just the keychain name.

[mbevc1]

Any reason not to go with latest v0.13.0 here?
https://github.com/gofrs/flock/releases

[mbevc1]

I wonder if some of those constants could be either exposed as configurable flags (e.g., --parallel-safe-timeout) or defined as named constants with a comment explaining the rationale for each value.

[mbevc1]

This seems like a lazy initialization via nil-checks on struct fields and it's harder to test and reason about than explicit construction. The tests inject mocks into the struct fields directly, which is fine, but production code paths rely on ensureSessionDependencies() being called. Consider using a constructor function or sync.Once to make initialization explicit and safe. At minimum, add a comment warning that this must be called before any field use.

[mbevc1]

The inner err = waiter.sleepAfterMiss(ctx) shadows the lock err variable. While this doesn't cause a bug here, it's confusing. Use a new variable: if sleepErr := waiter.sleepAfterMiss(ctx); sleepErr != nil.