Skip to content

feat(fips): FIPS 140-3 build variant (arc-fips)#512

Merged
xe-nvdk merged 6 commits into
mainfrom
feat/fips-build
Jun 21, 2026
Merged

feat(fips): FIPS 140-3 build variant (arc-fips)#512
xe-nvdk merged 6 commits into
mainfrom
feat/fips-build

Conversation

@xe-nvdk

@xe-nvdk xe-nvdk commented Jun 21, 2026

Copy link
Copy Markdown
Member

Summary

Tier 2 defense-grade work, item 1 of 4: an optional arc-fips build for US defense/federal and other regulated environments requiring validated cryptography.

  • FIPS is a build variant, not a version. Same source at the same version (26.06.2); built with -tags=duckdb_arrow,fips + GOFIPS140=v1.0.0 (the CMVP-certified Go Cryptographic Module snapshot), running GODEBUG=fips140=only (baked in). FIPS-ness lives in the artifact name (arc-fips-*, image :VERSION-fips), never the version string.
  • Covers OSS + Enterprise — one arc-fips binary; Enterprise FIPS = arc-fips + license key.
  • Standard build is unchanged (except the long-standing TLS 1.2 floor, now pinned explicitly).

What runs through the FIPS module

  • Token hashing: bcrypt → PBKDF2-HMAC-SHA256 (stdlib crypto/pbkdf2). Legacy bcrypt/sha256 hashes verify in the default build; the fips build fails them closed with a "rotate this token" warning (bcrypt is not linked into the fips binary — CI enforces this via an import-graph check).
  • HKDF: x/crypto/hkdf → stdlib crypto/hkdf (inside the FIPS boundary). Arg order preserved; a known-answer test pins the derived key byte-for-byte so a cluster split can't slip through.
  • TLS: TLS 1.2 floor in both builds; fips build additionally restricts cipher suites + curves to the FIPS-approved set (drops X25519). Standard build keeps Go's defaults. MQTT refuses InsecureSkipVerify in the fips build. The fips API listener replaces Fiber's opaque ListenTLS so Arc owns the negotiated suites.
  • Startup fail-closed: the fips binary refuses to start if not actually in FIPS mode; logs fips_mode in the banner.

Build & release

  • make build-fips / test-fips / fips-check; Dockerfile FIPS=1 arg.
  • release-build.yml: build-binaries matrix gains a variant dimension (standard/fips × amd64/arm64) → signed arc-fips-* binaries + bundles; SLSA provenance covers them; new docker-build-fips job pushes a cosign-signed :VERSION-fips image to GHCR + Docker Hub. CI verifies no x/crypto/bcrypt|hkdf in the fips build and that it boots with fips_mode=true.

CMVP framing (important)

The Go Cryptographic module is CMVP-certified (v1.0.0). Arc itself is not a CMVP-listed module — this is not a claim that "Arc is FIPS 140-3 validated." Docs and release notes state this precisely.

Operator cutover

Moving an existing deployment to arc-fips requires rotating API tokens once (existing bcrypt hashes fail closed; new tokens are PBKDF2). Documented in release notes + the FIPS guide.

Test plan

  • go build + go test pass in both variants (-tags=duckdb_arrow and ...,fips)
  • gofmt/go vet clean on changed files, both variants
  • HKDF known-answer test confirms byte-identical key across the stdlib swap
  • PBKDF2 round-trip + malformed-fails-closed + legacy fail-closed (fips) / legacy-verifies (default) tests
  • make fips-check (import-graph) clean; standard build still pulls bcrypt (backward compat)
  • Real arc-fips binary smoke-booted: fips_mode=true, reaches "Arc is ready!" (CGO + GOFIPS140 coexist)
  • Internal review: config matrix + deep reviewer (3 findings fixed) + adversarial fix-check (caught & fixed a latent TLS-1.2-floor regression)
  • Test release dry-run (workflow_dispatch test_mode=true) to verify arc-fips-* artifacts + :VERSION-fips image

Docs

Release notes (RELEASE_NOTES_2026.06.2.md), arc README.md, enterprise product-definition updated here. User-facing guide (configuration/fips.md) lands in the docs.basekick.net repo.

🤖 Generated with Claude Code

Tier 2 defense-grade work, item 1: an optional FIPS build of Arc for US
defense/federal and other regulated environments. Same source/version as the
standard build — FIPS is a build variant, not a separate version line.

Crypto changes (apply to both builds where noted):
- Token hashing: bcrypt -> PBKDF2-HMAC-SHA256 (FIPS-approved, stdlib
  crypto/pbkdf2). Self-describing $pbkdf2-sha256$ format, constant-time compare.
  Legacy bcrypt/sha256 hashes verify in the default build; the fips build fails
  them closed with a "rotate this token" warning (bcrypt is not linked into the
  fips binary at all).
- HKDF: golang.org/x/crypto/hkdf -> stdlib crypto/hkdf (inside the FIPS
  boundary). Argument order preserved; a known-answer test pins the derived key
  byte-for-byte so a cluster split cannot slip through.
- TLS: explicit TLS 1.2 floor in both builds (pre-existing behavior, now pinned
  so it survives Go default changes and ignores GODEBUG=tls10server). The fips
  build additionally restricts cipher suites + curves to the FIPS-approved set
  (drops X25519); the standard build keeps Go's defaults unchanged. MQTT refuses
  InsecureSkipVerify in the fips build.
- API listener: the fips build replaces Fiber's opaque ListenTLS with a hardened
  tls.NewListener so Arc owns the negotiated suites; the default build is
  unchanged.

Runtime/build:
- New internal/fips package: Enabled() (runtime), BuildTagged (compile-time,
  fips/!fips tagged), HardenTLSConfig().
- arc-fips bakes GODEBUG=fips140=only (//go:debug) and fails closed at startup
  if not actually in FIPS mode. Startup banner logs fips_mode.
- Built with GOFIPS140=v1.0.0 (the CMVP-certified Go Cryptographic Module
  snapshot). Same duckdb_arrow performance tag as the standard build.

Build/CI:
- Makefile: build-fips, test-fips, fips-check (import-graph lint).
- Dockerfile: FIPS=1 build arg.
- release-build.yml: build-binaries matrix gains a variant dimension
  (standard/fips x amd64/arm64) producing signed arc-fips-* binaries; SLSA
  provenance covers them; new docker-build-fips job pushes a cosign-signed
  :VERSION-fips image to GHCR + Docker Hub. CI verifies the fips build links no
  x/crypto/bcrypt|hkdf and boots with fips_mode=true.

Docs: release notes + arc README + enterprise product-definition. User guide
lands in docs.basekick.net. Honest CMVP framing throughout: the Go module is
certified; Arc itself is not a CMVP-listed module.
@xe-nvdk

xe-nvdk commented Jun 21, 2026

Copy link
Copy Markdown
Member Author

@gemini-code-assist please review

Focus areas for this FIPS 140-3 build-variant PR:

  • Build-tag correctness: fips/!fips files in internal/fips/ and internal/auth/auth_hash_*.go — does the default build behave identically to before, and the fips build link no x/crypto/bcrypt?
  • TLS posture: HardenTLSConfig applies a TLS 1.2 floor in both builds but FIPS ciphers/curves only in the fips build (standard build keeps Go defaults incl. X25519). Verify no behavior change for standard-build users beyond the pre-existing 1.2 floor.
  • HKDF migration: stdlib crypto/hkdf.Key(hash, secret, salt, info, len) preserves the exact secret/salt order vs the old x/crypto/hkdf.New — a flip silently splits the cluster. KAT test pins the bytes.
  • PBKDF2 token hashing: parser robustness (no panic/bypass on malformed $pbkdf2-sha256$ input), constant-time compare, and the legacy fail-closed semantics in the fips build.
  • CI: the build-binaries variant matrix, docker-build-fips cosign signing, and the import-graph + runtime fips_mode verification steps.

Comment thread internal/auth/auth_hash_legacy.go Dismissed

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an optional FIPS 140-3 build variant (arc-fips) compiled against the CMVP-certified Go Cryptographic Module. Key changes include migrating API token hashing from bcrypt to PBKDF2-HMAC-SHA256, transitioning cluster replication key derivation to the standard library's crypto/hkdf, and enforcing FIPS-approved TLS configurations (cipher suites and curves) for the API, cluster, and MQTT paths. Feedback on the changes highlights a missing NextProtos configuration in the custom FIPS TLS listener, which disables HTTP/2 negotiation and could lead to performance regressions.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread internal/api/server.go

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a FIPS 140-3 build variant (arc-fips) for Arc, migrating token hashing to PBKDF2, restricting TLS configurations to FIPS-approved primitives, and utilizing the standard library's crypto/hkdf. Feedback on these changes highlights a critical security issue where the deprecated CurvePreferences field is ignored in Go 1.23+, allowing non-approved curves like X25519 to persist; this must be updated to CurveIDs along with its associated tests. Additionally, the PBKDF2 verification logic should enforce upper bounds on iteration counts and salt/hash lengths to prevent potential CPU and memory denial-of-service attacks, and the fips-check Makefile target should be hardened to prevent silent success if the underlying go list command fails.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread internal/fips/fips.go
Comment thread internal/auth/auth.go
Comment thread internal/fips/fips_test.go
Comment thread internal/fips/fips_test.go
Comment thread Makefile
Gemini round 2 + CodeQL. Two valid findings hardened, three declined with
evidence.

PBKDF2 verify hardening (Gemini security-high + adversarial fix-check):
- Upfront pbkdf2MaxEncodedLen (128) length guard BEFORE any Atoi/base64 decode.
  The adversarial check showed the original bounds validated semantic length
  only AFTER decoding — a 100MB base64 field decodes to ~78MB before rejection,
  and (verify is lazy on the persisted SQLite row) re-triggers on every matching
  auth. Guard input size first, then parse.
- iter ceiling (pbkdf2MaxIterations = 10M) + salt/key length pins.
- Raft validateTokenEntry caps TokenHash (<=512) and TokenPrefix (<=256) so a
  rogue cluster-proposed entry never persists into the FSM + SQLite.
- Tests: abusive iter, wrong-length salt, multi-MB hash all fail closed; FSM
  rejects an oversized hash.

API listener (Gemini medium): set NextProtos ["http/1.1"]. Declined the
suggested "h2" — Fiber's engine (fasthttp) is HTTP/1.1-only and Fiber's own
ListenTLS sets no h2 ALPN, so advertising h2 would let a client negotiate a
protocol the server cannot speak.

Makefile fips-check (Gemini medium): capture stdout+stderr and check go list's
exit status, so a compile/list failure fails the target instead of silently
printing OK.

Declined (verified):
- CurveIDs (Gemini security-high, x3): hallucinated. tls.Config.CurveIDs does
  not exist in Go 1.26 (unknown field); CurvePreferences is correct, not
  deprecated, and the stdlib doc states setting it restricts curves.
- CodeQL #235 (weak SHA256, auth_hash_legacy.go): pre-existing verify-only
  backward-compat path relocated unchanged; dismissed won't-fix.
@xe-nvdk

xe-nvdk commented Jun 21, 2026

Copy link
Copy Markdown
Member Author

Addressed review round 2 (Gemini + CodeQL) in 14545a5. Two findings accepted and hardened, three declined with evidence.

✅ Accepted

PBKDF2 verify DoS (security-high)internal/auth/auth.go + internal/cluster/raft/fsm.go

  • Added an upfront pbkdf2MaxEncodedLen (128) length guard before any Atoi/base64.Decode. An adversarial check on the first fix showed the bounds validated semantic length only after decoding — a 100 MB base64 field decodes to ~78 MB before the len != expected check rejects it, and since verify is lazy on the persisted SQLite row it re-triggers on every matching auth. Now: bound input size first, then parse.
  • iter ceiling (pbkdf2MaxIterations = 10_000_000) + salt/key length pins.
  • validateTokenEntry now caps TokenHash (≤512) and TokenPrefix (≤256) so a rogue cluster-proposed entry never persists into the FSM/SQLite in the first place.
  • Tests added: abusive iter, wrong-length salt, multi-MB hash all fail closed; FSM rejects an oversized hash.

Makefile fips-check masks go list failure (medium) — now captures stdout+stderr and checks the exit status; a compile/list failure fails the target instead of silently printing OK.

API listener ALPN (medium) — set NextProtos: ["http/1.1"].

❌ Declined (verified)

CurveIDs instead of CurvePreferences (security-high, ×3) — respectfully, this is incorrect for our toolchain. tls.Config.CurveIDs does not exist in Go 1.26 (unknown field CurveIDs in struct literal of type tls.Config — confirmed by compiling it). CurvePreferences is not deprecated/ignored: the Go 1.26 stdlib doc states "If empty, the default will be used... To disable it, set CurvePreferences explicitly." So the existing code correctly restricts curves (drops X25519) in the FIPS build. Adopting the suggestion would break the build.

NextProtos: ["h2", "http/1.1"] — declined the h2 part. Fiber's engine (fasthttp) is HTTP/1.1-only and Fiber's own ListenTLS sets no h2 ALPN, so advertising h2 would let a client negotiate a protocol the server can't speak. Set ["http/1.1"] instead.

CodeQL #235 (weak SHA256, auth_hash_legacy.go:29) — dismissed won't-fix. This is a pre-existing verify-only backward-compat path (was in auth.go before this PR, relocated unchanged into the !fips file); new tokens are PBKDF2 and the fips build removes the path entirely.

@gemini-code-assist please review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an optional FIPS 140-3 build variant (arc-fips) compiled against the CMVP-certified Go Cryptographic Module. It updates the build system, centralizes FIPS posture and TLS hardening, migrates token hashing from bcrypt to standard library PBKDF2, and transitions replication key derivation to standard library HKDF. Feedback on the changes identifies two critical compilation errors in internal/auth/auth.go where the token string is passed directly to pbkdf2.Key instead of being converted to a byte slice.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread internal/auth/auth.go
Comment thread internal/auth/auth.go
@xe-nvdk

xe-nvdk commented Jun 21, 2026

Copy link
Copy Markdown
Member Author

Round 3 — both "critical compilation error" findings on pbkdf2.Key are declined: they're incorrect for our toolchain, and the suggested change would actually break the build.

This PR uses the stdlib crypto/pbkdf2 (Go 1.24+), whose signature is:

func Key(h func() hash.Hash, password string, salt []byte, iter, keyLength int) ([]byte, error)

The password argument is a string, so passing token directly is correct. This differs from the older golang.org/x/crypto/pbkdf2.Key, which took []byte — that appears to be the signature the suggestion is based on.

Compiler evidence (Go 1.26):

  • Current code (pbkdf2.Key(sha256.New, token, salt, ...)) — compiles; the whole tree builds in both variants (go build ./cmd/... ./internal/... and -tags=fips).
  • The suggested []byte(token) form — does not compile: cannot use []byte(token) (value of type []byte) as string value in argument to pbkdf2.Key.

CI on this PR builds arc and arc-fips for amd64+arm64 and runs the test suite, so a real compile error here would fail the run.

No change made for these two. (This is the same shape as the round-2 CurveIDs finding — a confident claim about a stdlib API signature that doesn't match Go 1.24+/1.26.) Everything from round 2 (PBKDF2 DoS bounds, ALPN, fips-check) remains as committed in 14545a5.

Ignacio Van Droogenbroeck added 4 commits June 20, 2026 20:13
…coverage

Token validation (HIGH):
- applyRotateToken previously accepted NewHash/NewPrefix with only a non-empty
  check, bypassing the length caps validateTokenEntry enforces on create/restore.
  A rogue cluster node could rotate a token to a multi-megabyte hash that lands
  in every node's FSM + SQLite and forces large allocations on each matching
  auth verify. Extracted a shared validateTokenHashAndPrefix (maxTokenHashLen
  512, maxTokenPrefixLen 256) and applied it on BOTH the create and rotate paths
  so they cannot drift. Tests: rotate rejects an oversized hash and leaves the
  original unchanged.

FIPS CI/release coverage:
- FIPS Docker image smoke test: docker-build-fips now boots the pushed
  :VERSION-fips image and asserts "fips_mode":true (catches packaging defects
  the binary-level checks can't).
- FIPS image Trivy scan: vuln-scan attaches a FIPS-image JSON report.
- FIPS image SBOM: sbom emits a FIPS container SPDX (source SBOM is shared —
  identical Go module graph).
- FIPS .deb/.rpm packages: debian-build + rpm-build gain a variant matrix
  producing arc-fips packages that Conflicts/Provides/Replaces (deb) and
  Conflicts/Obsoletes/Provides (rpm) the standard arc package — same
  /usr/bin/arc + arc.service, so an operator installs exactly one. Coexistence
  directives inserted via a targeted sed to avoid blank-line surgery on the
  spec/control body.

Release notes updated for the new FIPS packages, SBOM, and Trivy artifacts.
The internal/fips package doc pointed at a non-existent "fips_enabled.go". The
//go:debug fips140=only directive actually lives in cmd/arc/fips.go (package
main, where //go:debug takes effect). Comment-only.
The FIPS-build legacy-hash rejection fired a Warn on every auth attempt that
matched a legacy-hashed token's prefix. At Warn this (a) spams logs under
repeated auth and (b) acts as a token-existence oracle for anyone observing
logs — "token not found" is silent while "found but legacy, rejected" logged.
Drop to Debug: still available when actively diagnosing a failed auth, but not
emitted into normal log output. No behavior change (still fails closed).
…oS bounds

- Legacy-token rejection is logged at debug (matches the code change), not as a
  warning.
- Document the PBKDF2 verify bounds (length guard before decode, iteration cap,
  salt/key length pins) and the create+rotate length caps on the cluster
  replication path.
@xe-nvdk xe-nvdk merged commit eb362f9 into main Jun 21, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants