Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/developer-manual/develop-guideline.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,10 @@ Everything else — `controller-gen`, `kustomize`, `setup-envtest`, `golangci-li
downloaded into the repository's `bin/` directory by the Makefile on first use. Do not install those
globally; a mismatched version is a source of failures that do not reproduce in CI.

Kubernetes 1.29 is the supported floor.
End-to-end tests run against the Kubernetes versions listed in
[Kubernetes](../user-manual/environment/kubernetes.md#supported-versions). Note that
`KIND_K8S_VERSION` in each Makefile is only the default for a local cluster and is older than what
CI exercises — CI overrides it with its own matrix.

## Repository layout

Expand Down
2 changes: 1 addition & 1 deletion docs/operators/spark-k8s-operator.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ them.

## Prerequisites

- Kubernetes 1.29+
- Kubernetes — see [supported versions](../user-manual/environment/kubernetes.md#supported-versions)
- kubectl
- Helm v3+ — see [Installation](../quick-start/installation.md)
- An S3-compatible object store holding the event logs, reachable from the cluster
Expand Down
127 changes: 126 additions & 1 deletion docs/user-manual/environment/kubernetes.md
Original file line number Diff line number Diff line change
@@ -1 +1,126 @@
# Kubernetes environment
---
title: Kubernetes
---

Kubedoop runs on any conformant Kubernetes cluster. This page covers the versions it is tested
against and the handful of cluster capabilities the built-in Operators depend on.

## Supported versions

Every Operator's end-to-end suite runs against the same matrix on each pull request and each
release:

| Kubernetes | Status |
|------------|--------|
| 1.35 | Tested |
| 1.34 | Tested |
| 1.33 | Tested |

Other versions are not exercised by CI. Older releases may well work — nothing in the Operators
deliberately requires a recent API — but if you run one, you are the first to find out.

The Helm charts declare no `kubeVersion` constraint, so Helm will not stop you installing on
anything. Treat the table above as the supported range rather than a limit the tooling enforces.

## Cluster requirements

### CSI

Two of the built-in Operators are CSI drivers rather than ordinary controllers:

| Operator | CSI driver | Provides |
|----------|------------|----------|
| `secret-operator` | `secrets.kubedoop.dev` | Delivers credentials and certificates into pods as volumes |
| `listener-operator` | `listeners.kubedoop.dev` | Exposes pods and reports the address back into the pod |

Both register a `CSIDriver` object and run a node component as a DaemonSet, so the cluster must
allow CSI drivers and let those pods mount the kubelet directory. A managed cluster that forbids
privileged workloads outright will not be able to run them, and without them almost nothing else
works — every product cluster gets its secrets and its service exposure through these two.

`secret-operator`'s driver is registered with `attachRequired: false`, `podInfoOnMount: true` and
both `Ephemeral` and `Persistent` volume lifecycle modes.

### The kubelet directory

The node components mount the kubelet's plugin directories from the host. The chart default is:

```yaml
kubeletDir: /var/lib/kubelet
```

**Distributions that put the kubelet somewhere else need this overridden**, or the driver never
registers and every pod that wants a secret volume stays `Pending`. microk8s and k3s are the usual
cases:

```bash
helm install secret-operator kubedoop/secret-operator -n operators \
--set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet
```

Check where your distribution keeps it before installing:

```bash
ps aux | grep kubelet | grep -o '\-\-root-dir=[^ ]*'
```

If that prints nothing, the default is in use.

### Storage

Products that keep state — HDFS, Kafka, ZooKeeper — request PersistentVolumeClaims, so the cluster
needs a working `StorageClass`. Most managed clusters ship one; a bare kind cluster has
`standard` from the local-path provisioner.

```bash
kubectl get storageclass
```

`listener-operator` additionally creates its own StorageClass, `listeners.listeners.kubedoop.dev`,
used by the ephemeral volumes that carry listener addresses. That one is installed for you.

### RBAC

The Operators create and manage StatefulSets, Services, ConfigMaps, PodDisruptionBudgets and their
own custom resources. Installing via Helm creates the ClusterRoles they need, so you need
permission to create cluster-scoped RBAC at install time.

## Local clusters

[kind](https://kind.sigs.k8s.io/) is what the project's own end-to-end tests use, and it is the
quickest way to try Kubedoop:

```bash
kind create cluster --image kindest/node:v1.35.0
```

minikube and k3d work too. On k3d, remember the kubelet directory note above.

Size the cluster for what you deploy rather than for Kubedoop itself: the Operators are small, but a
three-node ZooKeeper ensemble plus HDFS plus a query engine will not fit in a default single-node
kind cluster without raising its resources.

## Verifying a cluster

Before installing, confirm the basics:

```bash
kubectl version # server version within the supported range
kubectl get storageclass # at least one, ideally a default
kubectl auth can-i create clusterrole # needed for the Helm install
```

After installing the built-in Operators, check that the CSI drivers registered:

```bash
kubectl get csidrivers
```

Both `secrets.kubedoop.dev` and `listeners.kubedoop.dev` should be listed. If they are missing, the
kubelet directory is the first thing to check.

## Related

- [Installation](../../quick-start/installation.md)
- [Service Discovery](../../core-concepts/connectivity/service-discovery.md)
- [Resource Management](../../core-concepts/resources/resource-manage.md)
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,9 @@ Kubedoop 不是单个仓库,各部分都在
由 Makefile 在首次使用时下载到仓库的 `bin/` 目录。不要全局安装它们,
版本不一致会导致本地失败而 CI 无法复现(反之亦然)。

Kubernetes 支持的最低版本是 1.29。
端到端测试运行在 [Kubernetes](../user-manual/environment/kubernetes.md#支持的版本) 页列出的版本上。
注意各 Makefile 里的 `KIND_K8S_VERSION` 只是本地集群的默认值,比 CI 实际覆盖的版本旧——
CI 会用自己的矩阵覆盖它。

## 仓库结构

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ title: Spark

## 前置条件

- Kubernetes 1.29+
- Kubernetes —— 参见[支持的版本](../user-manual/environment/kubernetes.md#支持的版本)
- kubectl
- Helm v3+ —— 参见[安装](../quick-start/installation.md)
- 一个集群内可访问、存放事件日志的 S3 兼容对象存储
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1,120 @@
# Kubernetes 环境
---
title: Kubernetes
---

Kubedoop 可以运行在任何符合规范的 Kubernetes 集群上。本页说明它经过测试的版本,
以及内置 Operator 依赖的少数几项集群能力。

## 支持的版本

每个 Operator 的端到端测试都会在每次 PR 和每次发版时跑同一套版本矩阵:

| Kubernetes | 状态 |
|------------|------|
| 1.35 | 已测试 |
| 1.34 | 已测试 |
| 1.33 | 已测试 |

其他版本 CI 不会覆盖。更老的版本很可能也能用——Operator 并没有刻意依赖新的 API——
但真要跑,你就是第一个发现问题的人。

Helm chart 没有声明 `kubeVersion` 约束,因此 Helm 不会阻止你装在任何版本上。
请把上表当作受支持的范围,而不是工具强制执行的限制。

## 集群要求

### CSI

内置 Operator 中有两个是 CSI 驱动,而不是普通的控制器:

| Operator | CSI 驱动 | 提供 |
|----------|----------|------|
| `secret-operator` | `secrets.kubedoop.dev` | 以卷的形式把凭据和证书送进 Pod |
| `listener-operator` | `listeners.kubedoop.dev` | 暴露 Pod,并把地址回写进 Pod |

两者都会注册 `CSIDriver` 对象,并以 DaemonSet 形式运行节点组件,因此集群必须允许 CSI 驱动,
并允许这些 Pod 挂载 kubelet 目录。完全禁止特权工作负载的托管集群跑不了它们,
而缺了它们几乎什么都用不了——每个产品集群的密钥分发和服务暴露都要经过这两个 Operator。

`secret-operator` 的驱动注册参数为 `attachRequired: false`、`podInfoOnMount: true`,
并同时支持 `Ephemeral` 和 `Persistent` 两种卷生命周期模式。

### kubelet 目录

节点组件需要从宿主机挂载 kubelet 的插件目录。chart 默认值为:

```yaml
kubeletDir: /var/lib/kubelet
```

**把 kubelet 放在别处的发行版必须覆盖这个值**,否则驱动永远注册不上,
所有需要密钥卷的 Pod 都会卡在 `Pending`。microk8s 和 k3s 是最常见的情况:

```bash
helm install secret-operator kubedoop/secret-operator -n operators \
--set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet
```

安装前先确认你的发行版把它放在哪:

```bash
ps aux | grep kubelet | grep -o '\-\-root-dir=[^ ]*'
```

没有输出说明用的是默认路径。

### 存储

有状态的产品——HDFS、Kafka、ZooKeeper——会申请 PersistentVolumeClaim,
因此集群需要一个可用的 `StorageClass`。大多数托管集群自带;
裸的 kind 集群带有 local-path provisioner 提供的 `standard`。

```bash
kubectl get storageclass
```

`listener-operator` 还会额外创建自己的 StorageClass `listeners.listeners.kubedoop.dev`,
供承载监听器地址的临时卷使用。这个不需要你操心。

### RBAC

Operator 需要创建和管理 StatefulSet、Service、ConfigMap、PodDisruptionBudget 以及它们自己的
自定义资源。通过 Helm 安装时会创建所需的 ClusterRole,因此安装者需要具备创建集群级 RBAC 的权限。

## 本地集群

[kind](https://kind.sigs.k8s.io/) 是项目自身端到端测试所使用的环境,也是试用 Kubedoop 最快的方式:

```bash
kind create cluster --image kindest/node:v1.35.0
```

minikube 和 k3d 同样可用。使用 k3d 时请注意上面关于 kubelet 目录的说明。

集群规格应当按你要部署的产品来定,而不是按 Operator 本身:Operator 很轻,
但三节点 ZooKeeper 加 HDFS 再加一个查询引擎,塞不进未调整资源的默认单节点 kind 集群。

## 检查集群

安装前先确认基础条件:

```bash
kubectl version # 服务端版本在受支持范围内
kubectl get storageclass # 至少有一个,最好有默认的
kubectl auth can-i create clusterrole # Helm 安装需要
```

安装完内置 Operator 后,确认 CSI 驱动已注册:

```bash
kubectl get csidrivers
```

`secrets.kubedoop.dev` 和 `listeners.kubedoop.dev` 都应当出现在列表里。
如果缺失,首先要排查的就是 kubelet 目录。

## 相关内容

- [安装](../../quick-start/installation.md)
- [服务发现](../../core-concepts/connectivity/service-discovery.md)
- [资源管理](../../core-concepts/resources/resource-manage.md)
Loading