rebase rivian fork 2.10.3 with main. - #1
Open
Codecheetos2023 wants to merge 5687 commits into
Open
Conversation
|
approved |
Avoid reading `c.gw.cfg.TLSConfig` while unlocked Signed-off-by: Daniele Sciascia <daniele@nats.io>
…af remote is removed (#8304) When a solicited leafnode remote with `jetstream_cluster_migrate` disconnects, `checkJetStreamMigrate` moves the account's R>1 stream/consumer raft nodes into observer mode, and the only place that state was reverted was after a successful reconnect. Removing (2.14) or disabling (2.12) the remote through a configuration reload left those raft nodes as observers until a server restart.
…#8305) Two fixes for atomic batch publishing in `processJetStreamAtomicBatchMsg`: - Fix a batch of exactly `max_batch_size` messages being rejected with `JSAtomicPublishTooLargeBatchError` when committed via EOB, since the marker carries `seq+1` but is not itself stored. - Fix the R1 (non-clustered) commit path storing the raw staged message verbatim instead of the transformed subject/headers/payload produced by `checkMsgHeadersPreClusteredProposal`, which left counter messages with empty payloads and permanently broke the counter.
Avoid reading `c.gw.cfg.TLSConfig` while unlocked
…xiedConns Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
…checks Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
…ale /varz leaf remotes (#8308) Fixes three 2.14 bugs: - **Scheduler**: `getScheduledMessages` formatted the `Nats-Schedule-Next` header with whole-second `RFC3339` precision, causing `@every` schedules with sub-second intervals (e.g. `@every 1500ms`) to collapse to whole-second cadence after the first fire. - **Fast batch**: a commit message failing `checkMsgHeadersPreClusteredProposal` in `gapOk` mode returned early without completing the commit, stranding the batch. - **Monitoring**: `updateVarzConfigReloadableFields` never rebuilt `varz.LeafNode.Remotes`, so the cached `/varz` endpoint kept serving the pre-reload leafnode remotes list after a config reload.
Signed-off-by: uwezkhan <uwezkhan053@gmail.com>
Slice bounds panic out of decodePeerState (server/raft.go) on a peer
state whose trailing id is short:
panic: runtime error: slice bounds out of range [:8] with capacity 3
server.decodePeerState(...) raft.go:4792
The per-peer loop guard was `ri < len(buf)`, so it entered the body
whenever any byte remained and then sliced a fixed 8-byte id with
`buf[ri:ri+idLen]`. EntryPeerState data from a peer reaches this decoder
directly, so a body whose last id is shorter than idLen reads past the
buffer.
Before: a peer state with fewer than idLen bytes left for the next id
read out of bounds and panicked.
After: the guard is `ri+idLen <= len(buf)`, so a partial id leaves
knownPeers short of expectedPeers and the existing `len(ps.knownPeers)
!= expectedPeers` check returns errCorruptPeers, which callers already
handle.
Tradeoff: a body that is correctly framed but a whole id short still
decodes the ids it has and is rejected by the count check, same as
today; the only behavioral change is the truncated-id case moving from
panic to errCorruptPeers. Sibling decoders (decodeAppendEntry,
decodeAppendEntryResponse, decodeVoteRequest, decodeVoteResponse) use
exact-length checks and are unchanged.
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
`fs.Compact` executed compression and encryption in the wrong order, and missed the `CompressionInfo` header.
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
…outes This change addresses a regression where service import replies were dropped when the requester and responder were on different cluster nodes. The fix enables delivery of response service imports that arrive via a route to be sent to route subscriptions, ensuring proper message handling in a multi-node environment. Signed-off-by: Sina Darbouy <sinadarbouy@gmail.com>
Closes #8272 ## Summary - Clear cached external authorization when refreshed account claims no longer enable external auth. - Add regression coverage for auth users, allowed accounts, and xkey state being removed by an account claim update. ## Testing - go test ./server -run TestAuthCallout -count=1 - golangci-lint run --timeout=5m --config=.golangci.yml ./server - codex review --base upstream/main
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Fix an off-by-one in `applyCommit` where the entry at exactly `n.applied` (snapshot's last index) could be re-applied after a snapshot install, resulting in a WAL reset. Relates to #8320
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Michael Röschter <michael@roeschter.de> Dialtimeout in leafnodes section Signed-off-by: Michael Röschter michael@roeschter.de Dialtimeout in leafnodes section Signed-off-by: Michael Röschter michael@roeschter.de Added unit test for dial_timeout default and configuration precedence Signed-off-by: Michael Röschter <michael@roeschter.de>
Fixes #8426 Tentative fix - there is no tooling in the project to unit test network delay during connection setup (raw packet delay). Signed-off-by: Michael Röschter <michael@roeschter.de>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Follow-up of #8413 An idempotent stream create was not actually idempotent. The `Created` timestamp in the assignment would be updated to the current time, instead of preserving what was in the assignment already. This could result in recreating the stream and losing its data if such an "idempotent" stream create was performed while a node that hosts it was down and it then came back up and caught up from a meta snapshot containing the updated `Created` timestamp. Only the `Created` timestamp is meant to be used for detecting a recreated stream/consumer. However, since the result of this bug may exist in meta logs for versions 2.14.4 and 2.12.14, this additional handling ensures the stream is only deleted when both the group name and the created time are different.
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
`syncBlocks` should leave all message blocks synced after it returns. This was not always the case when a block needed compaction during the same pass. Specifically, if the block was initially marked as not needing sync, i.e. `mb.needSync = false`, then `syncBlocks` would miss the fact that `compactWithFloor` could later set `mb.needSync = true` on that same block. Keep track of blocks compacted and sync both the new compacted file and its directory before returning. Signed-off-by: Daniele Sciascia <daniele@nats.io>
Signed-off-by: Daniele Sciascia <daniele@nats.io>
Found via Antithesis testing: `upgradeRouteToSolicited` could upgrade a gossip-dialed connection to `Explicit` while keeping its gossip URL, so after a disconnect `routeStillValid` rejected that URL and the reconnect was silently abandoned, leaving a route pool slot unfilled between two live servers. This PR adopts the configured URL during the upgrade, so reconnects always pass `routeStillValid`.
`syncBlocks` should leave all message blocks synced after it returns. This was not always the case when a block needed compaction during the same pass. Specifically, if the block was initially marked as not needing sync, i.e. `mb.needSync = false`, then `syncBlocks` would miss the fact that `compactWithFloor` could later set `mb.needSync = true` on that same block. Keep track of blocks compacted and sync both the new compacted file and its directory before returning.
Signed-off-by: Neil Twigg <neil@nats.io>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
…d origin Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Do single directory fsync after all blocks have been processed, instead of syncing the dir multiple times if more than one block was compacted. Signed-off-by: Daniele Sciascia <daniele@nats.io>
Signed-off-by: Daniele Sciascia <daniele@nats.io>
Do single directory fsync after all blocks have been processed, instead of syncing the dir multiple times if more than one block was compacted.
Previously if you started sourcing from a stream that was recreated, it would wait for the recreated stream's sequence to move back up to the last sourced sequence before it accepted new messages. This PR detects a stream being recreated by noticing a changed `Nats-Stream-Identity`, which is the hash of the stream's created time. There exists no unique identifier for a stream/consumer lifetime apart from the created time, so that's used and is the most practical since collisions are unlikely enough. - This hash is appended to the source header to know what "identity" of the stream was associated with the highest sequence for that source. (source header before: `TEST 1 > > foo`, after: `TEST 1 > > foo sLkPMPK6`) - This identity is passed in the consumer create request to prevent a consumer from being created based on sequences derived from the previous stream. An error is returned (`JSConsumerStreamIdentityMismatch`) if a mismatch is detected, the request is then retried with reset sequences to allow sourcing from the beginning. - The consumer reset request does not allow for the identity to be passed in, and instead relies on the caller to notice a different identity and subsequent in-memory state reset. This prevents unnecessary churn of requests since the reset will always happen based on a consumer that already exists and needs to be reset anyway. - Both the consumer create and reset responses contain this identity to get informed of the latest identity of the stream, which is then passed into the source header. This PR additionally requires that the starting sequence for a consumer is respected, even for sourcing consumers that were previously (incorrectly) clamped within the sourced stream's first/last sequence range. Upgrade note: the servers on both ends will need to be updated for this mechanism to kick in. A downgrade should ignore the extra appended identity from the header and fall back to the previous behavior. Resolves #8346, #6206, #5847
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
… group Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
…eted Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
…atchup (#8460) This PR fixes two remaining issues, and removes the final two FIXMEs: - Migrations would add peers as fast as possible, without checking whether the peer being added was even online. Adding an offline peer grows the quorum size, which could drop the group out of quorum until enough nodes came online. `selectPeerToAdd` now prefers the candidate we've heard from most recently, and only adds an unheard peer if the live members still reach quorum in the grown group. - Migrations also didn't wait for upper-layer stream catchup to complete before removing old peers. An old peer is now only removed if the group left after the removal retains a store-current majority, based on `Current`/`Lag` and the stream's active `catchupPeers`. Follow-up of #8432
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #NNNgit pull --rebase origin main)Resolves #
Changes proposed in this pull request: