Add Kafka DC-DR docs#944
Conversation
![kodiak-appscode[bot]](https://avatars.githubusercontent.com/in/92481?s=60&v=4){kind=small}
📝 WalkthroughWalkthroughThis PR adds Kafka DC-DR documentation across an index page, conceptual overview, detailed user guide, and operational runbook. It covers topology, deployment, fencing, switchover/failback behavior, observability, failure scenarios, cleanup, and stated limitations. ChangesKafka DC-DR Documentation
Estimated code review effort: 2 (Simple) | ~15 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (7)
docs/guides/kafka/dr/guide/index.md (2)
21-23: 📐 Maintainability & Code Quality | 🔵 TrivialVerify cross-reference link paths.
Absolute paths
/docs/guides/kafka/dr/overview/index.mdand/docs/guides/kafka/dr/runbook/index.mdmay not work correctly if the site uses a baseURL with a path prefix. Consider Hugorelrefshortcodes or relative paths.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/dr/guide/index.md` around lines 21 - 23, The cross-reference links in the Kafka DR guide use absolute paths that may break when the site is served under a non-root baseURL. Update the references in the guide content to use Hugo-safe linking for the DC-DR Overview and DC-DR Runbook, such as relref or equivalent relative links, so the paths resolve correctly regardless of deployment prefix.
25-25: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDescriptive link text needed.
"Please start here" uses non-descriptive link text. Replace with "the KubeDB getting started guide".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/dr/guide/index.md` at line 25, The link text in the KubeDB docs intro is too generic and should be made descriptive for accessibility. Update the markdown in the guide so the link pointing to /docs/README.md uses the phrase “the KubeDB getting started guide” instead of “here”, keeping the surrounding sentence and link target unchanged.Source: Linters/SAST tools
docs/guides/kafka/dr/overview/index.md (2)
30-30: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDescriptive link text needed.
"Please start here" uses non-descriptive link text. Replace with something like "the KubeDB getting started guide".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/dr/overview/index.md` at line 30, The markdown link in the KubeDB intro callout uses non-descriptive text, so update the link in the overview content to use a clearer label such as the KubeDB getting started guide while keeping the same target. Locate the change in the introductory paragraph near the “New to KubeDB?” message and adjust the link text in that markdown snippet.Source: Linters/SAST tools
24-28: 📐 Maintainability & Code Quality | 🔵 TrivialVerify cross-reference link paths.
The absolute paths
/docs/guides/kafka/dr/guide/index.mdand/docs/guides/kafka/dr/runbook/index.mdmay break if the site is mounted at a sub-path or uses a different baseURL. Consider using Hugorelrefor relative paths (e.g.,../guide/or{{< relref "guide" >}}) for portability across environments.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/dr/overview/index.md` around lines 24 - 28, The overview page uses absolute links in the DC-DR reference list, which can break under a non-root base URL. Update the links in the overview content to use Hugo-friendly portable references such as relref or relative paths, and verify the existing entries for the guide and runbook still point to the correct target pages.docs/guides/kafka/dr/runbook/index.md (3)
87-87: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMissing word in phase transition description.
"
phasemovesFailingOvertoSteady" → "phasemoves fromFailingOvertoSteady".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/dr/runbook/index.md` at line 87, The phase transition sentence is missing the word “from” in the description. Update the wording in the runbook text so the phase transition is described as moving from FailingOver to Steady, keeping the existing survivor writable behavior unchanged.
15-15: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueHyphenate compound modifier.
"cross data center" modifies "disaster recovery"; use "cross-data-center disaster recovery" for clarity.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/dr/runbook/index.md` at line 15, The runbook wording uses a compound modifier incorrectly in the Kafka DR description; update the phrase in the relevant markdown content to use “cross-data-center disaster recovery” instead of “cross data center” so the modifier is hyphenated consistently. Locate the text near the scenario overview in the runbook and revise the sentence accordingly.
23-23: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse descriptive link text.
Replace "here" with descriptive text, e.g., "See the KubeDB getting started guide."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/dr/runbook/index.md` at line 23, The link text in the runbook intro is non-descriptive and should be replaced with meaningful text. Update the markdown in the opening sentence to use a descriptive label for the /docs/README.md link, following the existing phrasing near “New to KubeDB?” so the destination is clear without relying on “here”.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@docs/guides/kafka/dr/guide/index.md`:
- Around line 21-23: The cross-reference links in the Kafka DR guide use
absolute paths that may break when the site is served under a non-root baseURL.
Update the references in the guide content to use Hugo-safe linking for the
DC-DR Overview and DC-DR Runbook, such as relref or equivalent relative links,
so the paths resolve correctly regardless of deployment prefix.
- Line 25: The link text in the KubeDB docs intro is too generic and should be
made descriptive for accessibility. Update the markdown in the guide so the link
pointing to /docs/README.md uses the phrase “the KubeDB getting started guide”
instead of “here”, keeping the surrounding sentence and link target unchanged.
In `@docs/guides/kafka/dr/overview/index.md`:
- Line 30: The markdown link in the KubeDB intro callout uses non-descriptive
text, so update the link in the overview content to use a clearer label such as
the KubeDB getting started guide while keeping the same target. Locate the
change in the introductory paragraph near the “New to KubeDB?” message and
adjust the link text in that markdown snippet.
- Around line 24-28: The overview page uses absolute links in the DC-DR
reference list, which can break under a non-root base URL. Update the links in
the overview content to use Hugo-friendly portable references such as relref or
relative paths, and verify the existing entries for the guide and runbook still
point to the correct target pages.
In `@docs/guides/kafka/dr/runbook/index.md`:
- Line 87: The phase transition sentence is missing the word “from” in the
description. Update the wording in the runbook text so the phase transition is
described as moving from FailingOver to Steady, keeping the existing survivor
writable behavior unchanged.
- Line 15: The runbook wording uses a compound modifier incorrectly in the Kafka
DR description; update the phrase in the relevant markdown content to use
“cross-data-center disaster recovery” instead of “cross data center” so the
modifier is hyphenated consistently. Locate the text near the scenario overview
in the runbook and revise the sentence accordingly.
- Line 23: The link text in the runbook intro is non-descriptive and should be
replaced with meaningful text. Update the markdown in the opening sentence to
use a descriptive label for the /docs/README.md link, following the existing
phrasing near “New to KubeDB?” so the destination is clear without relying on
“here”.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: d2f2c0a9-c50d-4738-b9e0-77ddd05f1d12
📒 Files selected for processing (4)
docs/guides/kafka/dr/_index.mddocs/guides/kafka/dr/guide/index.mddocs/guides/kafka/dr/overview/index.mddocs/guides/kafka/dr/runbook/index.md
|
Visit the preview URL for this PR (updated for commit 32f4366): https://kubedb-v2-hugo--pr944-kafka-dc-dr-docs-tx5v0sq7.web.app (expires Wed, 08 Jul 2026 03:13:32 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: 0f29ae8ae0bd54a99bf2b223b6833be47acd5943 |
Signed-off-by: Tamal Saha <tamal@appscode.com>
tamalsaha
force-pushed
the
kafka-dc-dr-docs
branch
from
32f4366 to
c555664
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/kafka/dr/guide/index.md`:
- Line 25: The markdown link text is non-descriptive (“here”) in the new-user
prompt and the same accessibility issue also appears in the Overview and Runbook
sections. Update the link labels in this guide to use meaningful text that
describes the destination, and make the changes in the relevant markdown entries
rather than relying on generic anchor text.
- Around line 256-293: The produce-fence description is inconsistent: it
currently sounds like only the non-active cluster is fenced, but the switchover
steps in the Kafka DR guide refer to closing the active cluster’s fence and
opening the target’s. Update the wording in this section to state that every
Member DC has a produce fence with two states: open when it holds the Lease and
closed when it does not, and that switchover simply moves the fence between
clusters. Make the same terminology consistent wherever this model is described
so operators see one clear state model.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 130bb446-9be5-4383-8a61-acbe8cd6dc12
📒 Files selected for processing (4)
docs/guides/kafka/dr/_index.mddocs/guides/kafka/dr/guide/index.mddocs/guides/kafka/dr/overview/index.mddocs/guides/kafka/dr/runbook/index.md
✅ Files skipped from review due to trivial changes (1)
- docs/guides/kafka/dr/_index.md
| architecture, and the [DC-DR Runbook](/docs/guides/kafka/dr/runbook/index.md) for | ||
| scenario-by-scenario procedures. | ||
|
|
||
| > **New to KubeDB?** Please start [here](/docs/README.md). |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use descriptive link text.
"here" is not descriptive for screen-reader users and fails accessibility standards. Same issue exists in Overview (line 30) and Runbook (line 23).
📝 Suggested fix
-> **New to KubeDB?** Please start [here](/docs/README.md).
+> **New to KubeDB?** Please start with the [KubeDB introduction](/docs/README.md).📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| > **New to KubeDB?** Please start [here](/docs/README.md). | |
| > **New to KubeDB?** Please start with the [KubeDB introduction](/docs/README.md). |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 25-25: Link text should be descriptive
(MD059, descriptive-link-text)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/kafka/dr/guide/index.md` at line 25, The markdown link text is
non-descriptive (“here”) in the new-user prompt and the same accessibility issue
also appears in the Overview and Runbook sections. Update the link labels in
this guide to use meaningful text that describes the destination, and make the
changes in the relevant markdown entries rather than relying on generic anchor
text.
| A non-active cluster must refuse producer writes, fail closed: a cluster that cannot | ||
| confirm it holds the Lease denies produce. There are two fence mechanisms. | ||
|
|
||
| - **ACL fence (authorization on):** revoke produce ACLs for the CLIENT principals on | ||
| the non-active cluster. This requires Kafka authorization enabled (the | ||
| `StandardAuthorizer`, set only when security is on). | ||
| - **Listener-gate fence (default posture):** gate the client listener on the non-active | ||
| cluster so producers cannot connect. The shipped examples default to | ||
| `disableSecurity: true`, and a cluster with no authorization has no ACLs to revoke, | ||
| so the listener gate is the default-posture fence. | ||
|
|
||
| DC-DR therefore requires one of the two: auth-on for the ACL fence, or the listener | ||
| gate. Two rules keep the fence from breaking replication: | ||
|
|
||
| - **Never fence the MM2 connector principal or `super.users`.** The fence must target | ||
| only client principals. Fencing the connector principal (or the operator's auth-secret | ||
| `super.users` user) would break mirroring and consumer-offset sync along with client | ||
| produce. | ||
| - **Do not blanket-mirror ACLs.** `MirrorSourceConnector` defaults | ||
| `sync.topic.acls.enabled=true`, which would copy the active cluster's client produce | ||
| ACLs onto the standby and re-grant produce there, undoing the fence. The operator sets | ||
| `sync.topic.acls.enabled=false` and manages client ACLs per cluster. | ||
|
|
||
| ## Planned switchover (drained, zero record loss) | ||
|
|
||
| Move the active DC on purpose by annotating the Kafka: | ||
|
|
||
| ```bash | ||
| $ kubectl annotate kafka -n demo kf-dcdr dr.kubedb.com/switchover-to=dc-b | ||
| ``` | ||
|
|
||
| The hub then: | ||
|
|
||
| 1. checks the target is a known, healthy DC within the MM2 lag budget; | ||
| 2. sets `phase: FailingOver` and quiesces producers by closing the active cluster's | ||
| produce fence; | ||
| 3. waits for MM2 to drain to near-zero lag, so the target holds every record; | ||
| 4. flips the bootstrap endpoint to the target, opens the target's fence, and reverses |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Clarify the produce fence state model.
The "Produce fence" section describes the fence as applying only to "the non-active cluster" (lines 259, 262), but the switchover procedure then describes "closing the active cluster's produce fence" (line 291) and "opens the target's fence" (line 293). This implies the fence exists on every cluster with two states, which contradicts the earlier description and will confuse operators.
Clarify that the fence is a gate on every Member DC: open (writes allowed) when the cluster holds the Lease, and closed (fail-closed, writes denied) when it does not. During switchover, the orchestrator closes the active cluster's fence and opens the target's.
Same phrasing appears in the Overview (lines 281-283) and Runbook (lines 135-137).
📝 Suggested rewording
A non-active cluster must refuse producer writes, fail closed: a cluster that cannot
confirm it holds the Lease denies produce. There are two fence mechanisms.
+The fence is a gate on every Member DC: **open** (writes allowed) on the Lease
+holder, and **closed** (fail-closed) on all others.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| A non-active cluster must refuse producer writes, fail closed: a cluster that cannot | |
| confirm it holds the Lease denies produce. There are two fence mechanisms. | |
| - **ACL fence (authorization on):** revoke produce ACLs for the CLIENT principals on | |
| the non-active cluster. This requires Kafka authorization enabled (the | |
| `StandardAuthorizer`, set only when security is on). | |
| - **Listener-gate fence (default posture):** gate the client listener on the non-active | |
| cluster so producers cannot connect. The shipped examples default to | |
| `disableSecurity: true`, and a cluster with no authorization has no ACLs to revoke, | |
| so the listener gate is the default-posture fence. | |
| DC-DR therefore requires one of the two: auth-on for the ACL fence, or the listener | |
| gate. Two rules keep the fence from breaking replication: | |
| - **Never fence the MM2 connector principal or `super.users`.** The fence must target | |
| only client principals. Fencing the connector principal (or the operator's auth-secret | |
| `super.users` user) would break mirroring and consumer-offset sync along with client | |
| produce. | |
| - **Do not blanket-mirror ACLs.** `MirrorSourceConnector` defaults | |
| `sync.topic.acls.enabled=true`, which would copy the active cluster's client produce | |
| ACLs onto the standby and re-grant produce there, undoing the fence. The operator sets | |
| `sync.topic.acls.enabled=false` and manages client ACLs per cluster. | |
| ## Planned switchover (drained, zero record loss) | |
| Move the active DC on purpose by annotating the Kafka: | |
| ```bash | |
| $ kubectl annotate kafka -n demo kf-dcdr dr.kubedb.com/switchover-to=dc-b | |
| ``` | |
| The hub then: | |
| 1. checks the target is a known, healthy DC within the MM2 lag budget; | |
| 2. sets `phase: FailingOver` and quiesces producers by closing the active cluster's | |
| produce fence; | |
| 3. waits for MM2 to drain to near-zero lag, so the target holds every record; | |
| 4. flips the bootstrap endpoint to the target, opens the target's fence, and reverses | |
| A non-active cluster must refuse producer writes, fail closed: a cluster that cannot | |
| confirm it holds the Lease denies produce. There are two fence mechanisms. | |
| The fence is a gate on every Member DC: **open** (writes allowed) on the Lease | |
| holder, and **closed** (fail-closed) on all others. | |
| - **ACL fence (authorization on):** revoke produce ACLs for the CLIENT principals on | |
| the non-active cluster. This requires Kafka authorization enabled (the | |
| `StandardAuthorizer`, set only when security is on). | |
| - **Listener-gate fence (default posture):** gate the client listener on the non-active | |
| cluster so producers cannot connect. The shipped examples default to | |
| `disableSecurity: true`, and a cluster with no authorization has no ACLs to revoke, | |
| so the listener gate is the default-posture fence. | |
| DC-DR therefore requires one of the two: auth-on for the ACL fence, or the listener | |
| gate. Two rules keep the fence from breaking replication: | |
| - **Never fence the MM2 connector principal or `super.users`.** The fence must target | |
| only client principals. Fencing the connector principal (or the operator's auth-secret | |
| `super.users` user) would break mirroring and consumer-offset sync along with client | |
| produce. | |
| - **Do not blanket-mirror ACLs.** `MirrorSourceConnector` defaults | |
| `sync.topic.acls.enabled=true`, which would copy the active cluster's client produce | |
| ACLs onto the standby and re-grant produce there, undoing the fence. The operator sets | |
| `sync.topic.acls.enabled=false` and manages client ACLs per cluster. | |
| ## Planned switchover (drained, zero record loss) | |
| Move the active DC on purpose by annotating the Kafka: | |
🧰 Tools
🪛 LanguageTool
[style] ~281-~281: Try using a descriptive adverb here.
Context: ..., zero record loss) Move the active DC on purpose by annotating the Kafka: ```bash $ kub...
(ON_PURPOSE_DELIBERATELY)
🪛 markdownlint-cli2 (0.22.1)
[warning] 284-284: Dollar signs used before commands without showing output
(MD014, commands-show-output)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/kafka/dr/guide/index.md` around lines 256 - 293, The
produce-fence description is inconsistent: it currently sounds like only the
non-active cluster is fenced, but the switchover steps in the Kafka DR guide
refer to closing the active cluster’s fence and opening the target’s. Update the
wording in this section to state that every Member DC has a produce fence with
two states: open when it holds the Lease and closed when it does not, and that
switchover simply moves the fence between clusters. Make the same terminology
consistent wherever this model is described so operators see one clear state
model.
1 participant
Adds cross data center disaster recovery (DC-DR) documentation for KubeDB Kafka under
docs/guides/kafka/dr, mirroring the four-file structure of the MongoDB DR docs but rewritten for Kafka's design.New pages:
_index.md: the DR section menu entry.overview/index.md: conceptual overview, why Kafka is its own DR camp (no single primary), the five core rules, DC roles table, the single-CR/single-endpoint model, prerequisites, a deploy walkthrough (PlacementPolicy + distributed Kafka), observingstatus.disasterRecovery, unplanned failover, planned switchover, failback, and cleanup.guide/index.md: components-and-where-they-run table, the DC-name contract, deploying and what the operator creates (per-DC Kafka clusters, per-DC ConnectClusters, the three MM2 connectors on the standby), connecting/producing, consumers resuming via translated offsets, monitoring MM2 lag, the produce fence (ACL vs listener-gate), switchover, failback, scaling, and day-2 ops.runbook/index.md: scenario-by-scenario procedures (intra-DC broker loss, full active-DC loss, clean DC-vs-DR partition, planned switchover, consumers not resuming, failback, standby loss, high MM2 lag, arbiter loss, coordination-plane outage, which DC is active, suspected split writes).Kafka DC-DR is active/passive: two self-contained Kafka clusters (one per Member DC, each with its own intra-DC KRaft quorum) joined by asynchronous MirrorMaker 2 mirroring, with the dr-controlplane Lease choosing the write cluster and a fail-closed produce fence preventing split writes. There is no zero-RPO unplanned failover and no rewind on failback.
Summary by CodeRabbit