From 71d5df39d690df2322e76cac87b979501f09af2c Mon Sep 17 00:00:00 2001 From: whg517 Date: Mon, 24 Aug 2026 16:18:51 +0800 Subject: [PATCH 1/2] docs(core-concepts): write the S3 page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces a one-line placeholder. Content is taken from the operator-go API rather than written from memory: pkg/apis/s3/v1alpha1 for the two CRDs and their defaults, pkg/s3/resolver.go for how inline/reference resolve and how the endpoint is assembled, pkg/s3/credentials.go for the secret key names, and hive-operator's ClusterConfigSpec for the product-side example. Covers S3Connection and S3Bucket, the inline-or-reference rule with its actual error strings, the ACCESS_KEY/SECRET_KEY contract and where the Secret Operator mounts it, TLS verification modes, and region. Gives pathStyle a section of its own. It defaults to false, which is right for AWS and wrong for MinIO and Ceph RGW, and nothing rejects the wrong value at admission — the resource applies, the pods start, and it fails on first object access. That is worth more than a table row. Co-Authored-By: Claude Opus 5 --- docs/core-concepts/resources/s3.md | 171 +++++++++++++++++- .../current/core-concepts/resources/s3.md | 161 ++++++++++++++++- 2 files changed, 329 insertions(+), 3 deletions(-) diff --git a/docs/core-concepts/resources/s3.md b/docs/core-concepts/resources/s3.md index 6d45a35d..8c19465e 100644 --- a/docs/core-concepts/resources/s3.md +++ b/docs/core-concepts/resources/s3.md @@ -1,4 +1,171 @@ +--- +title: S3 +--- -# S3 +Many Kubedoop products read and write object storage: Hive keeps table data there, Trino queries it, +Spark checkpoints to it. Rather than repeating endpoint and credential settings in every product +resource, Kubedoop models S3 as two cluster resources that products point at. -TODO +## Two objects + +| Object | Describes | +|--------|-----------| +| `S3Connection` | Where the object store is and how to reach it: host, port, TLS, region, addressing style, credentials | +| `S3Bucket` | A bucket name, plus the connection that bucket lives on | + +Both belong to the `s3.kubedoop.dev/v1alpha1` API group. + +Which one a product asks for depends on how it addresses storage. A product handed a whole bucket +takes an `S3Bucket`; a product that decides bucket names itself — Hive Metastore, for instance — +takes an `S3Connection` directly. + +## Inline or reference, never both + +Wherever Kubedoop accepts S3 configuration it accepts the same pair of fields: `inline` to define +the object in place, or `reference` to name an existing object in the same namespace. They are +mutually exclusive, and resolution fails if you set both or neither: + +```text +invalid S3 connection: inline and reference are mutually exclusive +invalid S3 connection: neither inline nor reference is set +``` + +Use `reference` as soon as a second product talks to the same object store, so the endpoint is +defined once. + +## S3Connection + +```yaml +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Connection +metadata: + name: minio +spec: + host: minio.default.svc.cluster.local + port: 9000 + pathStyle: true + region: us-east-1 + credentials: + secretClass: minio-credentials +``` + +| Field | Required | Default | Notes | +|-------|----------|---------|-------| +| `host` | yes | — | Hostname of the object store. No scheme; the scheme follows `tls` | +| `port` | no | — | Omitted entirely when unset, so the endpoint uses the scheme's default port | +| `credentials` | yes | — | SecretClass supplying the access keys, see below | +| `pathStyle` | no | `false` | Addressing style, see below | +| `tls` | no | — | Presence switches the endpoint to `https` | +| `region` | no | `us-east-1` | Signing region | + +The endpoint is assembled from these fields: the scheme is `https` when `tls` is set and `http` +otherwise, so there is no separate "use TLS" switch to keep in sync. + +## Credentials + +`credentials.secretClass` names a SecretClass that must publish exactly two keys: + +| Key | Read by products as | +|-----|---------------------| +| `ACCESS_KEY` | `AWS_ACCESS_KEY_ID` | +| `SECRET_KEY` | `AWS_SECRET_ACCESS_KEY` | + +The Secret Operator mounts them as files under `/kubedoop/secret//`, and products source +them into the environment: + +```bash +export AWS_ACCESS_KEY_ID="$(cat /kubedoop/secret/s3-credentials/ACCESS_KEY)" +export AWS_SECRET_ACCESS_KEY="$(cat /kubedoop/secret/s3-credentials/SECRET_KEY)" +``` + +Credentials may additionally carry a `scope`, which narrows what the issued credential covers — +see [Authentication](../security/authentication.md) for how SecretClass scopes work. + +## pathStyle: the field that breaks MinIO + +`pathStyle` selects how the bucket is addressed: + +| Value | Resulting URL | Correct for | +|-------|---------------|-------------| +| `false` (default) | `https://.` — virtual host | AWS S3 | +| `true` | `https:///` — path | MinIO, Ceph RGW, most self-hosted backends | + +The default is right for AWS and wrong for most in-cluster deployments. MinIO in particular serves +path-style only. With the default, the client resolves `.` — `warehouse.minio` for an +in-cluster MinIO — which does not exist in DNS. + +Nothing rejects this at admission. The resource applies cleanly, the pods start, and the failure +only surfaces on the first object access. **Set `pathStyle: true` for MinIO, Ceph RGW and similar +backends.** + +## TLS + +Setting `tls` switches the endpoint to `https`. The nested `verification` decides how the server +certificate is checked: + +```yaml +spec: + host: s3.example.com + tls: + verification: + server: + caCert: + secretClass: tls # a SecretClass that issues the CA certificate +``` + +| `verification` | Behaviour | +|----------------|-----------| +| `server.caCert.secretClass` | Verify against the CA published by that SecretClass | +| `server.caCert.webPki: {}` | Verify against the system's public CA bundle | +| `none: {}` | Do not verify the certificate | + +`none` disables verification entirely and should stay out of production. + +## S3Bucket + +An `S3Bucket` is a bucket name plus a connection, and the connection is itself an +inline-or-reference pair: + +```yaml +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Bucket +metadata: + name: warehouse +spec: + bucketName: warehouse + connection: + reference: minio +``` + +`bucketName` is the name on the object store; the resource's own `metadata.name` is only what other +Kubedoop resources refer to. They do not have to match. + +A bucket with no `connection` fails resolution: + +```text +invalid S3 bucket "warehouse": no connection is set +``` + +## Using it from a product + +Products expose the same inline-or-reference pair. Hive Metastore takes a connection under +`clusterConfig.s3`: + +```yaml +apiVersion: hive.kubedoop.dev/v1alpha1 +kind: HiveMetastore +metadata: + name: hive +spec: + clusterConfig: + s3: + reference: minio +``` + +Swapping `reference: minio` for an `inline:` block with the same fields as an `S3Connection` spec +is equivalent, and appropriate when only one product uses that endpoint. + +## Related + +- [Roles and role groups](../common-configuration-mechanisms/roles-and-role-groups.md) +- [Resource management](./resource-manage.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/resources/s3.md b/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/resources/s3.md index 07df9e2b..f33e70cb 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/resources/s3.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/resources/s3.md @@ -1,2 +1,161 @@ +--- +title: S3 +--- -# S3 +Kubedoop 中很多产品都要读写对象存储：Hive 把表数据放在上面，Trino 查询它，Spark 往里写检查点。 +为了不在每个产品资源里重复填写端点和凭据，Kubedoop 把 S3 抽象成两个集群资源，由产品去引用。 + +## 两个对象 + +| 对象 | 描述内容 | +|------|----------| +| `S3Connection` | 对象存储在哪、怎么连：host、port、TLS、region、寻址风格、凭据 | +| `S3Bucket` | 一个桶名，加上这个桶所在的连接 | + +两者都属于 `s3.kubedoop.dev/v1alpha1` API 组。 + +产品用哪一个，取决于它如何寻址存储。被交付整个桶的产品用 `S3Bucket`；自己决定桶名的产品—— +比如 Hive Metastore——直接用 `S3Connection`。 + +## inline 与 reference 二选一 + +凡是接受 S3 配置的地方，Kubedoop 都提供同一对字段：`inline` 就地定义，`reference` 引用同命名空间 +下已有对象的名字。两者互斥，同时设置或都不设置都会解析失败： + +```text +invalid S3 connection: inline and reference are mutually exclusive +invalid S3 connection: neither inline nor reference is set +``` + +只要有第二个产品要访问同一个对象存储，就该改用 `reference`，让端点只定义一次。 + +## S3Connection + +```yaml +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Connection +metadata: + name: minio +spec: + host: minio.default.svc.cluster.local + port: 9000 + pathStyle: true + region: us-east-1 + credentials: + secretClass: minio-credentials +``` + +| 字段 | 必填 | 默认值 | 说明 | +|------|------|--------|------| +| `host` | 是 | — | 对象存储主机名。不带协议头，协议由 `tls` 决定 | +| `port` | 否 | — | 不设置时端点里完全不带端口，走协议默认端口 | +| `credentials` | 是 | — | 提供访问密钥的 SecretClass，见下文 | +| `pathStyle` | 否 | `false` | 寻址风格，见下文 | +| `tls` | 否 | — | 一旦设置，端点切换为 `https` | +| `region` | 否 | `us-east-1` | 请求签名用的区域 | + +端点由这些字段拼装：设置了 `tls` 就用 `https`，否则用 `http`。因此不存在一个需要额外同步的 +"是否启用 TLS" 开关。 + +## 凭据 + +`credentials.secretClass` 指向的 SecretClass 必须提供且仅需提供两个键： + +| 键 | 产品读取为 | +|----|------------| +| `ACCESS_KEY` | `AWS_ACCESS_KEY_ID` | +| `SECRET_KEY` | `AWS_SECRET_ACCESS_KEY` | + +Secret Operator 会把它们以文件形式挂载到 `/kubedoop/secret//`，产品再注入到环境变量： + +```bash +export AWS_ACCESS_KEY_ID="$(cat /kubedoop/secret/s3-credentials/ACCESS_KEY)" +export AWS_SECRET_ACCESS_KEY="$(cat /kubedoop/secret/s3-credentials/SECRET_KEY)" +``` + +`credentials` 还可以带 `scope`，用于收窄签发凭据的适用范围——SecretClass 的作用域机制参见 +[认证](../security/authentication.md)。 + +## pathStyle：这个字段会让 MinIO 挂掉 + +`pathStyle` 决定桶的寻址方式： + +| 取值 | 生成的 URL | 适用于 | +|------|------------|--------| +| `false`（默认） | `https://.`——虚拟主机风格 | AWS S3 | +| `true` | `https:///`——路径风格 | MinIO、Ceph RGW 等绝大多数自建后端 | + +默认值对 AWS 是对的，对大多数集群内部署是错的。MinIO 尤其只支持路径风格。用默认值时客户端会去解析 +`.`——集群内的 MinIO 就是 `warehouse.minio`——这个域名在 DNS 里并不存在。 + +**准入阶段不会拦截这个错误。** 资源正常创建，Pod 正常启动，直到第一次访问对象时才失败。 +**MinIO、Ceph RGW 这类后端请设置 `pathStyle: true`。** + +## TLS + +设置 `tls` 会把端点切到 `https`。嵌套的 `verification` 决定如何校验服务端证书： + +```yaml +spec: + host: s3.example.com + tls: + verification: + server: + caCert: + secretClass: tls # 签发 CA 证书的 SecretClass +``` + +| `verification` | 行为 | +|----------------|------| +| `server.caCert.secretClass` | 用该 SecretClass 提供的 CA 校验 | +| `server.caCert.webPki: {}` | 用系统内置的公共 CA 集校验 | +| `none: {}` | 完全不校验证书 | + +`none` 会彻底关闭校验，不应出现在生产环境。 + +## S3Bucket + +`S3Bucket` 就是一个桶名加一个连接，而这个连接本身同样是 inline/reference 二选一： + +```yaml +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Bucket +metadata: + name: warehouse +spec: + bucketName: warehouse + connection: + reference: minio +``` + +`bucketName` 是对象存储上的真实桶名；资源自身的 `metadata.name` 只是其他 Kubedoop 资源引用它时 +用的名字。两者不必相同。 + +没有 `connection` 的桶会解析失败： + +```text +invalid S3 bucket "warehouse": no connection is set +``` + +## 在产品中使用 + +产品侧暴露的是同一对 inline/reference 字段。Hive Metastore 在 `clusterConfig.s3` 下接受一个连接： + +```yaml +apiVersion: hive.kubedoop.dev/v1alpha1 +kind: HiveMetastore +metadata: + name: hive +spec: + clusterConfig: + s3: + reference: minio +``` + +把 `reference: minio` 换成字段与 `S3Connection` spec 相同的 `inline:` 块，效果等价，适合只有一个 +产品使用该端点的场景。 + +## 相关内容 + +- [角色和角色组](../common-configuration-mechanisms/roles-and-role-groups.md) +- [资源管理](./resource-manage.md) From 3e5babe8059dc7aff24a0873384550425dbef747 Mon Sep 17 00:00:00 2001 From: whg517 Date: Mon, 24 Aug 2026 16:19:39 +0800 Subject: [PATCH 2/2] docs(core-concepts): write the pod placement and pod disruption pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both files were literally empty — 0 bytes in each language tree. Content comes from commons/v1alpha1 (pdb_types.go, config_types.go, cluster_types.go) and from the reconciler's affinity fold. Pod disruptions: the operator writes a PDB per role by default, so the page leads with "you are already protected". Documents that `enabled` is a pointer whose unset state means true — a role that names the block only to raise maxUnavailable keeps its PDB — that maxUnavailable defaults per product rather than to a fixed number, that roleConfig is role-scoped and role groups cannot override it, and that a PDB constrains voluntary disruptions only. Pod placement: the important part is that config.affinity replaces the layer beneath it wholesale rather than merging per member. Pinning a group with a nodeAffinity therefore discards the podAntiAffinity the product ships to spread a quorum, and nothing fails when it happens — the spec is valid and every status condition stays green. The page names the three layers in the reconciler's own words, quotes the AffinityOverridden warning event the operator emits, gives the kubectl command to find it, and shows restating the discarded member. Co-Authored-By: Claude Opus 5 --- .../operations/pod-disruptions.md | 86 +++++++++++ .../core-concepts/operations/pod-placement.md | 138 ++++++++++++++++++ .../operations/pod-disruptions.md | 82 +++++++++++ .../core-concepts/operations/pod-placement.md | 136 +++++++++++++++++ 4 files changed, 442 insertions(+) diff --git a/docs/core-concepts/operations/pod-disruptions.md b/docs/core-concepts/operations/pod-disruptions.md index e69de29b..95c3e566 100644 --- a/docs/core-concepts/operations/pod-disruptions.md +++ b/docs/core-concepts/operations/pod-disruptions.md @@ -0,0 +1,86 @@ +--- +title: Pod Disruptions +--- + +A Kubernetes cluster moves pods around for reasons that have nothing to do with your workload: +draining a node for an upgrade, rebalancing, scaling down the node pool. These are *voluntary* +disruptions, and a +[PodDisruptionBudget](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) is how a +workload tells the cluster how many of its pods may be taken down at once. + +This matters most for quorum-based roles. Evicting two of three ZooKeeper servers at the same time +does not slow the ensemble down — it loses quorum and stops. + +Kubedoop writes a PodDisruptionBudget **per role**, using a default the operator picks from what it +knows about the product. You do not have to configure anything to be protected. + +## Configuration + +Each role accepts a `roleConfig.podDisruptionBudget` block: + +```yaml +apiVersion: zookeeper.kubedoop.dev/v1alpha1 +kind: ZookeeperCluster +metadata: + name: zookeeper +spec: + server: + roleConfig: + podDisruptionBudget: + enabled: true + maxUnavailable: 1 + roleGroups: + default: + replicas: 3 +``` + +| Field | Type | Default | Meaning | +|-------|------|---------|---------| +| `enabled` | boolean | `true` | Whether the operator writes a PDB for this role | +| `maxUnavailable` | integer | product-specific | How many pods of the role may be down at once | + +Both fields are optional, and the block itself is optional. + +### `enabled` defaults to true, including when you omit it + +Leaving `podDisruptionBudget` out entirely, and writing a block that only sets `maxUnavailable`, +both leave the PDB enabled. A role that mentions the budget just to raise `maxUnavailable` still +gets its PDB — you only lose it by writing `enabled: false` explicitly. + +### `maxUnavailable` defaults per product + +When you do not set it, the operator supplies a value based on what it knows about the product +rather than a blanket number, because the safe count differs by role: a three-node quorum tolerates +one loss, a pool of stateless workers tolerates far more. + +Set it yourself only when you know the product's tolerance better than the operator does for your +topology. + +## Supplying your own PDB + +`enabled: false` stops the operator from writing a PDB for the role, which is what you want when +you intend to manage one yourself: + +```yaml +spec: + server: + roleConfig: + podDisruptionBudget: + enabled: false +``` + +The operator then leaves the role alone, and any PodDisruptionBudget you create with your own +selector applies normally. Note that turning it off and *not* replacing it leaves the role with no +protection against voluntary eviction at all. + +## What a PDB does not cover + +A PodDisruptionBudget constrains voluntary disruptions only. A node that hard-crashes, a kernel +panic, or a pod killed for exceeding its memory limit are all involuntary — the budget has no say. +Spreading a role across failure domains is the tool for that; see +[Pod placement](./pod-placement.md). + +## Related + +- [Pod placement](./pod-placement.md) +- [Roles and role groups](../common-configuration-mechanisms/roles-and-role-groups.md) diff --git a/docs/core-concepts/operations/pod-placement.md b/docs/core-concepts/operations/pod-placement.md index e69de29b..49f1601a 100644 --- a/docs/core-concepts/operations/pod-placement.md +++ b/docs/core-concepts/operations/pod-placement.md @@ -0,0 +1,138 @@ +--- +title: Pod Placement +--- + +Where a role's pods land decides how much a single failure costs. Three ZooKeeper servers on one +node are three pods that disappear together. Kubedoop products therefore ship a default +[affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) that spreads a +role across nodes, and let you replace it when your topology calls for something else. + +## The default is already there + +Every product declares affinity defaults for its own roles — typically a `podAntiAffinity` that +keeps replicas of a role off the same node. You do not need to write anything to get it. + +Read the section below before overriding it, because overriding does not mean adding. + +## Configuring affinity + +Affinity is set under `config`, which exists at two levels: + +```yaml +apiVersion: zookeeper.kubedoop.dev/v1alpha1 +kind: ZookeeperCluster +metadata: + name: zookeeper +spec: + server: + config: + affinity: # role level: applies to every group of this role + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/worker + operator: Exists + roleGroups: + default: + replicas: 3 + gpu: + replicas: 2 + config: + affinity: # role group level: applies to this group only + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: accelerator + operator: In + values: ["nvidia"] +``` + +The value is a standard Kubernetes +[`Affinity`](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling) +object and is passed through to the pod spec unchanged, so `nodeAffinity`, `podAffinity` and +`podAntiAffinity` all behave exactly as they do in a plain Deployment. + +## Affinity replaces, it does not merge + +There are three layers, from the bottom up: + +1. the product's role defaults +2. the role's `config.affinity` +3. the role group's `config.affinity` + +**Any layer that sets `affinity` replaces the layer beneath it entirely.** This follows the +Kubernetes rule and is what someone editing the field expects — but it has a consequence that is +easy to miss. + +Setting only `nodeAffinity` at the role group level, to pin a group to an instance type, discards +the product's `podAntiAffinity` along with it. The group is now pinned to the right nodes and no +longer spread across them, which is usually the opposite of what was intended. + +Nothing fails when this happens. The resource is valid, the pod spec is valid, and every status +condition stays green while the quorum quietly stops being spread. + +### How to tell it happened + +The operator emits a Warning event on the cluster resource: + +```text +Type Reason Message +Warning AffinityOverridden role "server" group "gpu": the role group's config replaces + config.affinity wholesale, discarding the podAntiAffinity declared + beneath it. config.affinity follows the Kubernetes rule and is not + merged per member; restate the discarded member alongside your own + to keep it +``` + +Check for it after changing affinity: + +```bash +kubectl get events --field-selector reason=AffinityOverridden +``` + +### Keeping what you replaced + +Restate the discarded member alongside your own: + +```yaml +config: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: accelerator + operator: In + values: ["nvidia"] + podAntiAffinity: # restated, or it is lost + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 70 + podAffinityTerm: + topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: zookeeper + app.kubernetes.io/instance: zookeeper + app.kubernetes.io/component: server +``` + +## Choosing a topology key + +`topologyKey` decides what "spread apart" means: + +| Key | Spreads across | +|-----|----------------| +| `kubernetes.io/hostname` | Nodes — protects against a single node failing | +| `topology.kubernetes.io/zone` | Availability zones — protects against a zone failing | + +Zone-level spreading needs enough nodes per zone to satisfy it. With +`requiredDuringSchedulingIgnoredDuringExecution` and fewer zones than replicas, the surplus pods +stay `Pending` forever; `preferredDuringSchedulingIgnoredDuringExecution` degrades instead of +blocking, which is why product defaults favour it. + +## Related + +- [Pod disruptions](./pod-disruptions.md) +- [Roles and role groups](../common-configuration-mechanisms/roles-and-role-groups.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/operations/pod-disruptions.md b/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/operations/pod-disruptions.md index e69de29b..04429350 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/operations/pod-disruptions.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/operations/pod-disruptions.md @@ -0,0 +1,82 @@ +--- +title: Pod 干扰 +--- + +Kubernetes 集群会因为和你的负载毫无关系的原因挪动 Pod：为升级排空节点、重新均衡、缩容节点池。 +这些属于**自愿性干扰**，而 +[PodDisruptionBudget](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) +正是负载用来告诉集群"我最多能同时下线几个 Pod"的手段。 + +对基于法定人数（quorum）的角色来说这尤其要紧。同时驱逐三个 ZooKeeper 节点中的两个，不是让集群变慢， +而是直接失去法定人数、停止服务。 + +Kubedoop 会**按角色**写出 PodDisruptionBudget，默认值由 Operator 根据它对该产品的了解选取。 +你不需要做任何配置就已经受到保护。 + +## 配置 + +每个角色都接受 `roleConfig.podDisruptionBudget` 配置块： + +```yaml +apiVersion: zookeeper.kubedoop.dev/v1alpha1 +kind: ZookeeperCluster +metadata: + name: zookeeper +spec: + server: + roleConfig: + podDisruptionBudget: + enabled: true + maxUnavailable: 1 + roleGroups: + default: + replicas: 3 +``` + +| 字段 | 类型 | 默认值 | 含义 | +|------|------|--------|------| +| `enabled` | 布尔 | `true` | Operator 是否为该角色写出 PDB | +| `maxUnavailable` | 整数 | 因产品而异 | 该角色允许同时下线的 Pod 数 | + +两个字段都是可选的，配置块本身也是可选的。 + +注意 `roleConfig` 是**角色级**配置，不会被下面的角色组继承或覆盖。 + +### `enabled` 默认为 true，省略时也是 + +完全不写 `podDisruptionBudget`，以及只写了 `maxUnavailable` 的配置块，两种情况下 PDB 都保持启用。 +一个角色仅仅为了调高 `maxUnavailable` 而提到这个块，它的 PDB 依然存在——只有显式写 +`enabled: false` 才会失去它。 + +### `maxUnavailable` 按产品取默认值 + +不设置时，Operator 会依据它对该产品的了解给出取值，而不是套用一个统一数字，因为安全阈值因角色而异： +三节点法定人数只能容忍损失一个，而无状态工作节点池能容忍的多得多。 + +只有当你比 Operator 更清楚该产品在你这套拓扑下的容忍度时，才需要自己设置。 + +## 使用自定义 PDB + +`enabled: false` 会让 Operator 不再为该角色写 PDB，适合你打算自己管理的场景： + +```yaml +spec: + server: + roleConfig: + podDisruptionBudget: + enabled: false +``` + +此后 Operator 不再干预该角色，你用自己的选择器创建的 PodDisruptionBudget 会正常生效。 +但要注意：关掉之后**不**补上替代品，等于该角色对自愿性驱逐完全失去保护。 + +## PDB 管不了的事 + +PodDisruptionBudget 只约束自愿性干扰。节点硬崩溃、内核 panic、Pod 因超出内存限制被杀， +这些都属于非自愿干扰，预算对它们没有发言权。应对这类情况的手段是把角色分散到不同故障域， +参见 [Pod 放置](./pod-placement.md)。 + +## 相关内容 + +- [Pod 放置](./pod-placement.md) +- [角色和角色组](../common-configuration-mechanisms/roles-and-role-groups.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/operations/pod-placement.md b/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/operations/pod-placement.md index e69de29b..0754292c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/operations/pod-placement.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/core-concepts/operations/pod-placement.md @@ -0,0 +1,136 @@ +--- +title: Pod 放置 +--- + +角色的 Pod 落在哪里，决定了一次故障的代价有多大。三个 ZooKeeper 节点都在同一台机器上，就是三个会 +一起消失的 Pod。因此 Kubedoop 的产品都自带一份默认 +[亲和性](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)配置， +把角色打散到不同节点；当你的拓扑需要别的安排时，也可以替换它。 + +## 默认配置已经存在 + +每个产品都为自己的角色声明了亲和性默认值——通常是一条让同一角色的副本避开同一节点的 +`podAntiAffinity`。你什么都不写就已经生效。 + +在覆盖它之前请先读下面一节，因为**覆盖不等于追加**。 + +## 配置亲和性 + +亲和性配置在 `config` 下，而 `config` 存在于两个层级： + +```yaml +apiVersion: zookeeper.kubedoop.dev/v1alpha1 +kind: ZookeeperCluster +metadata: + name: zookeeper +spec: + server: + config: + affinity: # 角色级：对该角色的所有角色组生效 + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/worker + operator: Exists + roleGroups: + default: + replicas: 3 + gpu: + replicas: 2 + config: + affinity: # 角色组级：只对该组生效 + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: accelerator + operator: In + values: ["nvidia"] +``` + +取值是标准的 Kubernetes +[`Affinity`](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling) +对象，会原样传递到 Pod spec，因此 `nodeAffinity`、`podAffinity`、`podAntiAffinity` +的行为与在普通 Deployment 中完全一致。 + +## 亲和性是替换，不是合并 + +一共三层，自下而上： + +1. 产品的角色默认值 +2. 角色的 `config.affinity` +3. 角色组的 `config.affinity` + +**任何一层只要设置了 `affinity`，就会整体替换它下面那一层。** 这遵循 Kubernetes 的惯例，也符合 +编辑该字段的人的预期——但它带来一个很容易忽略的后果。 + +在角色组级只写一个 `nodeAffinity` 把该组钉到某种机型上，会连同产品的 `podAntiAffinity` 一起丢弃。 +结果是这组 Pod 确实落到了正确的节点上，却不再彼此打散——通常与本意正好相反。 + +**发生这种情况时不会有任何失败。** 资源是合法的，Pod spec 是合法的，所有状态条件都是绿的， +而法定人数已经悄悄不再分散了。 + +### 如何发现 + +Operator 会在集群资源上发出 Warning 事件： + +```text +Type Reason Message +Warning AffinityOverridden role "server" group "gpu": the role group's config replaces + config.affinity wholesale, discarding the podAntiAffinity declared + beneath it. config.affinity follows the Kubernetes rule and is not + merged per member; restate the discarded member alongside your own + to keep it +``` + +改动亲和性之后建议检查一下： + +```bash +kubectl get events --field-selector reason=AffinityOverridden +``` + +### 保留被替换掉的部分 + +把被丢弃的成员和你自己的配置一起重新写出来： + +```yaml +config: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: accelerator + operator: In + values: ["nvidia"] + podAntiAffinity: # 必须重新声明，否则丢失 + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 70 + podAffinityTerm: + topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: zookeeper + app.kubernetes.io/instance: zookeeper + app.kubernetes.io/component: server +``` + +## 选择拓扑键 + +`topologyKey` 决定了"打散"是按什么维度： + +| 键 | 打散范围 | +|----|----------| +| `kubernetes.io/hostname` | 节点——防止单节点故障 | +| `topology.kubernetes.io/zone` | 可用区——防止单可用区故障 | + +按可用区打散要求每个可用区有足够的节点。如果用 +`requiredDuringSchedulingIgnoredDuringExecution` 而可用区数少于副本数，多出来的 Pod 会永远处于 +`Pending`；`preferredDuringSchedulingIgnoredDuringExecution` 则是降级而非阻塞， +这也是产品默认值偏向后者的原因。 + +## 相关内容 + +- [Pod 干扰](./pod-disruptions.md) +- [角色和角色组](../common-configuration-mechanisms/roles-and-role-groups.md)