From 8895cc4776a05e9fa8389b10beb11ff888cc1da6 Mon Sep 17 00:00:00 2001 From: whg517 Date: Mon, 24 Aug 2026 20:57:08 +0800 Subject: [PATCH] docs(operators): write the Spark operator page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First real page under operators/, filled from _template.md's section order. Content comes from the spark-k8s-operator repository: the CRD in api/v1alpha1, the port constants in internal/util, the cleaner constraints in internal/controller/historyserver/configmap.go, and a working manifest from test/e2e/smoke/submit-task. Leads with what the operator actually is. The repository name suggests Spark on Kubernetes generally, but the API has exactly one kind, SparkHistoryServer — there is no SparkApplication resource, and the operator does not submit or schedule jobs. Saying so up front is the single most useful thing this page does. Documents the single `node` role, the required clusterConfig.logFileDirectory.s3 bucket and prefix, listenerClass and what each value exposes, the three ports, OIDC authentication, and defaults taken from the CRD rather than guessed. Gives the event log cleaner its own section: it deletes from shared object storage, so the operator rejects `cleaner` on a role group with more than one replica, and rejects it at role level when the role has more than one role group. Note for future pages: MDX 3 parses `` as a JSX tag and fails the build. Use [text](url) form for bare URLs in docs pages. Co-Authored-By: Claude Opus 5 --- docs/operators/spark-k8s-operator.md | 336 +++++++++++++++++- .../current/operators/spark-k8s-operator.md | 326 ++++++++++++++++- 2 files changed, 659 insertions(+), 3 deletions(-) diff --git a/docs/operators/spark-k8s-operator.md b/docs/operators/spark-k8s-operator.md index d495d6e9..66d703f2 100644 --- a/docs/operators/spark-k8s-operator.md +++ b/docs/operators/spark-k8s-operator.md @@ -1,2 +1,336 @@ +--- +title: Spark +--- -# spark-k8s-operator +> Kubedoop Operator for Apache Spark — deploys and manages the Spark History Server on Kubernetes. + +## Overview + +[Apache Spark](https://spark.apache.org/) is a distributed engine for large-scale data processing. +When a Spark application runs it writes an *event log* describing every job, stage and task it +executed. Once the application finishes, its driver UI disappears along with it, and that event log +is all that remains. + +The **Spark History Server** reads those event logs back and serves the same web UI for applications +that have already completed. It is how you answer "why was last night's job slow?" after the job is +gone. + +This Operator manages the History Server. It does **not** submit or schedule Spark applications — +there is no `SparkApplication` resource in this API. You run Spark however you already do, point it +at an object store for event logs, and point a `SparkHistoryServer` at the same location to browse +them. + +## Prerequisites + +- Kubernetes 1.29+ +- kubectl +- Helm v3+ — see [Installation](../quick-start/installation.md) +- An S3-compatible object store holding the event logs, reachable from the cluster + +## Quick Start + +### Install the Operator + +Install the built-in Operators first — they are required: + +```bash +helm install commons-operator kubedoop/commons-operator -n operators --create-namespace +helm install listener-operator kubedoop/listener-operator -n operators +helm install secret-operator kubedoop/secret-operator -n operators +``` + +Then the Spark Operator: + +```bash +helm install spark-k8s-operator kubedoop/spark-k8s-operator -n operators +``` + +Verify it is running: + +```bash +kubectl get pods -n operators -l app.kubernetes.io/name=spark-k8s-operator +``` + +### Create a Namespace + +```bash +kubectl create ns spark +``` + +### Point at the event logs + +The History Server reads event logs from S3, so describe the bucket first. Credentials come from a +SecretClass, the connection from an `S3Connection`, and the bucket from an `S3Bucket`: + +```yaml +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Connection +metadata: + name: minio + namespace: spark +spec: + host: minio + port: 9000 + credentials: + secretClass: s3-credentials +--- +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Bucket +metadata: + name: spark-history + namespace: spark +spec: + bucketName: spark-history + connection: + reference: minio +``` + +> For the full set of connection options — TLS, region, and the `pathStyle` setting that most +> self-hosted backends need — see [S3](../core-concepts/resources/s3.md). + +### Deploy the History Server + +```bash +kubectl apply -f - < For more details, see +> [Roles and Role Groups](../core-concepts/common-configuration-mechanisms/roles-and-role-groups.md). + +### Configurations + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `clusterConfig.logFileDirectory.s3.bucket` | The bucket holding event logs, `inline` or `reference` | required | +| `clusterConfig.logFileDirectory.s3.prefix` | Path within the bucket | required | +| `clusterConfig.listenerClass` | How the UI is exposed | `cluster-internal` | +| `clusterConfig.authentication` | OIDC authentication for the UI | none | +| `clusterConfig.vectorAggregatorConfigMapName` | Vector aggregator discovery ConfigMap | none | +| `image.productVersion` | Spark version | `3.5.5` | +| `image.repo` | Image repository | `quay.io/zncdatadev` | +| `image.pullPolicy` | Image pull policy | `IfNotPresent` | +| `node.roleGroups..replicas` | Replicas in the group | `1` | +| `node.config.cleaner` | Enable event log cleanup | `false` | + +`clusterConfig` and `node` are both required; a resource without them is rejected. + +> Configuration can be set at the role level and overridden per role group — see +> [Overrides](../core-concepts/common-configuration-mechanisms/overrides.md). + +### Ports + +| Name | Port | Purpose | +|------|------|---------| +| `http` | 18080 | History Server web UI | +| `metrics` | 18081 | Metrics endpoint | +| `oidc` | 4180 | OIDC proxy, only when authentication is configured | + +### Listeners and Services + +`clusterConfig.listenerClass` decides how the UI is reachable: + +| Value | Exposure | +|-------|----------| +| `cluster-internal` (default) | ClusterIP — inside the cluster only | +| `external-unstable` | NodePort — address changes if the pod moves | +| `external-stable` | LoadBalancer — stable address | + +> For more details, see [Service Discovery](../core-concepts/connectivity/service-discovery.md). + +### Dependencies + +| Dependency | Required | Description | +|------------|----------|-------------| +| S3-compatible storage | Yes | Where event logs are read from; there is no other backend | +| `commons-operator`, `listener-operator`, `secret-operator` | Yes | Provide pod enrichment, service exposure and credential delivery | +| An OIDC provider | No | Only when `clusterConfig.authentication` is set | + +## Advanced + +### The event log cleaner + +Setting `cleaner: true` lets the History Server delete old event logs from the bucket. Because the +bucket is shared, **only one instance may clean**, and the Operator enforces that: + +```yaml +spec: + node: + roleGroups: + default: + replicas: 1 + config: + cleaner: true +``` + +Two configurations are rejected with an error rather than silently allowing several cleaners to race: + +- `cleaner` enabled on a role group whose `replicas` is greater than 1 +- `cleaner` enabled at the role level when the role has more than one role group + +If you need several role groups, enable the cleaner on exactly one single-replica group. + +### Resource Management + +```yaml +spec: + node: + config: + resources: + cpu: + min: "1" + max: "2" + memory: + limit: "4Gi" +``` + +> For more details, see [Resource Management](../core-concepts/resources/resource-manage.md). + +### Pod Placement + +Affinity is set under `config.affinity` on the role or a role group. Note that it **replaces** the +product's defaults rather than merging with them. + +> For more details, see [Pod Placement](../core-concepts/operations/pod-placement.md). + +### Authentication + +The UI can be put behind an OIDC provider by referencing an `AuthenticationClass`: + +```yaml +spec: + clusterConfig: + authentication: + authenticationClass: keycloak + oidc: + clientCredentialsSecret: spark-oidc-client + extraScopes: + - profile +``` + +The referenced Secret must contain `CLIENT_ID` and `CLIENT_SECRET`. When authentication is +configured the OIDC proxy port (4180) is added to the pod. + +> For more details, see [Authentication](../core-concepts/security/authentication.md). + +### Logging + +```yaml +spec: + node: + config: + logging: + containers: + node: + console: + level: INFO + file: + level: DEBUG +``` + +> For more details, see [Logging](../core-concepts/observability/logging.md). + +## Troubleshooting + +### Common Issues + +1. **The UI loads but lists no applications** + - **Symptom**: The History Server is running and reachable, but the application list is empty. + - **Cause**: `prefix` does not match the path your Spark applications write to, or the + applications are not writing event logs at all. + - **Resolution**: Confirm `spark.eventLog.enabled` is true in your applications and that + `spark.eventLog.dir` resolves to the same bucket and prefix. List the bucket to confirm files + are actually arriving. + +2. **The pod fails to start or cannot read the bucket** + - **Symptom**: The pod crash-loops, or the logs show S3 access errors. + - **Cause**: Wrong credentials, or virtual-host addressing against a backend that only serves + path-style. Nothing rejects the latter at apply time — it fails on first access. + - **Resolution**: Check that the SecretClass supplies `ACCESS_KEY` and `SECRET_KEY`, and set + `pathStyle: true` on the `S3Connection` for MinIO, Ceph RGW and similar backends. See + [S3](../core-concepts/resources/s3.md). + +3. **The resource is rejected when enabling the cleaner** + - **Symptom**: An error mentioning more than one role group or more than one replica with the + cleaner enabled. + - **Cause**: Several instances would compete to delete from the same bucket. + - **Resolution**: Enable `cleaner` on exactly one role group with `replicas: 1`. + +> For general operational topics, see +> [Pod Disruptions](../core-concepts/operations/pod-disruptions.md). + +## Clean Up + +```bash +kubectl delete sparkhistoryserver simple-history -n spark +kubectl delete ns spark +helm uninstall spark-k8s-operator -n operators +``` + +## Related Links + +- [Spark History Server documentation](https://spark.apache.org/docs/latest/monitoring.html) +- [Kubedoop Operator for Apache Spark on GitHub](https://github.com/zncdatadev/spark-k8s-operator) +- [Apache Spark on GitHub](https://github.com/apache/spark) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/operators/spark-k8s-operator.md b/i18n/zh/docusaurus-plugin-content-docs/current/operators/spark-k8s-operator.md index f0ee86fa..84905982 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/operators/spark-k8s-operator.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/operators/spark-k8s-operator.md @@ -1,4 +1,326 @@ +--- +title: Spark +--- -# spark-k8s-operator +> Kubedoop Operator for Apache Spark —— 在 Kubernetes 上部署和管理 Spark History Server。 -TODO +## 概述 + +[Apache Spark](https://spark.apache.org/) 是用于大规模数据处理的分布式引擎。Spark 应用运行时会写出 +**事件日志**（event log），记录它执行过的每个 job、stage 和 task。应用一旦结束，driver UI 也随之消失， +留下的就只有这份事件日志。 + +**Spark History Server** 负责把这些事件日志读回来，为已经结束的应用重新提供同样的 Web UI。 +当你想回答"昨晚那个任务为什么慢"时，靠的就是它。 + +本 Operator 管理的是 History Server。它**不负责**提交或调度 Spark 应用——该 API 中并没有 +`SparkApplication` 资源。你照旧用自己的方式运行 Spark，让它把事件日志写到对象存储， +再让 `SparkHistoryServer` 指向同一位置来浏览这些日志。 + +## 前置条件 + +- Kubernetes 1.29+ +- kubectl +- Helm v3+ —— 参见[安装](../quick-start/installation.md) +- 一个集群内可访问、存放事件日志的 S3 兼容对象存储 + +## 快速开始 + +### 安装 Operator + +先安装内置 Operator，它们是必需的： + +```bash +helm install commons-operator kubedoop/commons-operator -n operators --create-namespace +helm install listener-operator kubedoop/listener-operator -n operators +helm install secret-operator kubedoop/secret-operator -n operators +``` + +然后安装 Spark Operator： + +```bash +helm install spark-k8s-operator kubedoop/spark-k8s-operator -n operators +``` + +确认它已运行： + +```bash +kubectl get pods -n operators -l app.kubernetes.io/name=spark-k8s-operator +``` + +### 创建命名空间 + +```bash +kubectl create ns spark +``` + +### 指向事件日志 + +History Server 从 S3 读取事件日志，所以先把桶描述出来。凭据来自 SecretClass，连接来自 +`S3Connection`，桶来自 `S3Bucket`： + +```yaml +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Connection +metadata: + name: minio + namespace: spark +spec: + host: minio + port: 9000 + credentials: + secretClass: s3-credentials +--- +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Bucket +metadata: + name: spark-history + namespace: spark +spec: + bucketName: spark-history + connection: + reference: minio +``` + +> 完整的连接选项——TLS、region，以及大多数自建后端都需要的 `pathStyle`——参见 +> [S3](../core-concepts/resources/s3.md)。 + +### 部署 History Server + +```bash +kubectl apply -f - < 更多内容参见 +> [角色和角色组](../core-concepts/common-configuration-mechanisms/roles-and-role-groups.md)。 + +### 配置项 + +| 参数 | 说明 | 默认值 | +|------|------|--------| +| `clusterConfig.logFileDirectory.s3.bucket` | 存放事件日志的桶，`inline` 或 `reference` | 必填 | +| `clusterConfig.logFileDirectory.s3.prefix` | 桶内路径 | 必填 | +| `clusterConfig.listenerClass` | UI 的暴露方式 | `cluster-internal` | +| `clusterConfig.authentication` | UI 的 OIDC 认证 | 无 | +| `clusterConfig.vectorAggregatorConfigMapName` | Vector 聚合端发现用 ConfigMap | 无 | +| `image.productVersion` | Spark 版本 | `3.5.5` | +| `image.repo` | 镜像仓库 | `quay.io/zncdatadev` | +| `image.pullPolicy` | 镜像拉取策略 | `IfNotPresent` | +| `node.roleGroups..replicas` | 该组副本数 | `1` | +| `node.config.cleaner` | 是否启用事件日志清理 | `false` | + +`clusterConfig` 和 `node` 都是必填的，缺少任一的资源会被拒绝。 + +> 配置可以在角色级设置、并在角色组级覆盖——参见 +> [覆盖配置](../core-concepts/common-configuration-mechanisms/overrides.md)。 + +### 端口 + +| 名称 | 端口 | 用途 | +|------|------|------| +| `http` | 18080 | History Server Web UI | +| `metrics` | 18081 | 指标端点 | +| `oidc` | 4180 | OIDC 代理，仅在配置了认证时存在 | + +### 监听器与服务 + +`clusterConfig.listenerClass` 决定 UI 如何被访问： + +| 取值 | 暴露方式 | +|------|----------| +| `cluster-internal`（默认） | ClusterIP——仅集群内可访问 | +| `external-unstable` | NodePort——Pod 漂移后地址会变 | +| `external-stable` | LoadBalancer——地址稳定 | + +> 更多内容参见[服务发现](../core-concepts/connectivity/service-discovery.md)。 + +### 依赖 + +| 依赖 | 必需 | 说明 | +|------|------|------| +| S3 兼容存储 | 是 | 事件日志的唯一来源，没有其他后端可选 | +| `commons-operator`、`listener-operator`、`secret-operator` | 是 | 分别提供 Pod 信息补全、服务暴露和凭据分发 | +| OIDC 提供方 | 否 | 仅在设置了 `clusterConfig.authentication` 时需要 | + +## 进阶 + +### 事件日志清理器 + +设置 `cleaner: true` 可以让 History Server 删除桶中的旧事件日志。由于这个桶是共享的， +**只能有一个实例执行清理**，Operator 会强制这一点： + +```yaml +spec: + node: + roleGroups: + default: + replicas: 1 + config: + cleaner: true +``` + +以下两种配置会直接报错，而不是放任多个清理器互相竞争： + +- 在 `replicas` 大于 1 的角色组上启用 `cleaner` +- 角色拥有多个角色组时，在角色级启用 `cleaner` + +如果确实需要多个角色组，请只在其中一个单副本组上启用清理器。 + +### 资源管理 + +```yaml +spec: + node: + config: + resources: + cpu: + min: "1" + max: "2" + memory: + limit: "4Gi" +``` + +> 更多内容参见[资源管理](../core-concepts/resources/resource-manage.md)。 + +### Pod 放置 + +亲和性配置在角色或角色组的 `config.affinity` 下。注意它是**替换**产品默认值，而非与之合并。 + +> 更多内容参见[Pod 放置](../core-concepts/operations/pod-placement.md)。 + +### 认证 + +可以通过引用 `AuthenticationClass` 把 UI 置于 OIDC 提供方之后： + +```yaml +spec: + clusterConfig: + authentication: + authenticationClass: keycloak + oidc: + clientCredentialsSecret: spark-oidc-client + extraScopes: + - profile +``` + +被引用的 Secret 必须包含 `CLIENT_ID` 和 `CLIENT_SECRET`。配置认证后，Pod 上会增加 OIDC 代理端口 +（4180）。 + +> 更多内容参见[认证](../core-concepts/security/authentication.md)。 + +### 日志 + +```yaml +spec: + node: + config: + logging: + containers: + node: + console: + level: INFO + file: + level: DEBUG +``` + +> 更多内容参见[日志](../core-concepts/observability/logging.md)。 + +## 故障排查 + +### 常见问题 + +1. **UI 能打开，但没有任何应用** + - **现象**：History Server 正常运行也能访问，但应用列表是空的。 + - **原因**：`prefix` 与 Spark 应用实际写入的路径不一致，或者应用根本没有写事件日志。 + - **解决**：确认应用的 `spark.eventLog.enabled` 为 true，且 `spark.eventLog.dir` + 指向同一个桶和前缀。直接列一下桶，确认文件确实在写进来。 + +2. **Pod 启动失败或读不到桶** + - **现象**：Pod 反复重启，或日志里出现 S3 访问错误。 + - **原因**：凭据不对；或者对只支持路径风格的后端使用了虚拟主机寻址。后者在 apply 阶段不会被拦截， + 只会在首次访问时失败。 + - **解决**：确认 SecretClass 提供了 `ACCESS_KEY` 和 `SECRET_KEY`；对 MinIO、Ceph RGW + 这类后端，在 `S3Connection` 上设置 `pathStyle: true`。参见 + [S3](../core-concepts/resources/s3.md)。 + +3. **启用 cleaner 时资源被拒绝** + - **现象**：报错提示启用了 cleaner 的角色组数量或副本数大于 1。 + - **原因**：多个实例会争抢删除同一个桶中的内容。 + - **解决**：只在一个 `replicas: 1` 的角色组上启用 `cleaner`。 + +> 通用运维话题参见 [Pod 干扰](../core-concepts/operations/pod-disruptions.md)。 + +## 清理 + +```bash +kubectl delete sparkhistoryserver simple-history -n spark +kubectl delete ns spark +helm uninstall spark-k8s-operator -n operators +``` + +## 相关链接 + +- [Spark History Server 官方文档](https://spark.apache.org/docs/latest/monitoring.html) +- [Kubedoop Operator for Apache Spark（GitHub）](https://github.com/zncdatadev/spark-k8s-operator) +- [Apache Spark（GitHub）](https://github.com/apache/spark)