diff --git a/docs/operators/spark-k8s-operator.md b/docs/operators/spark-k8s-operator.md index d495d6e9..66d703f2 100644 --- a/docs/operators/spark-k8s-operator.md +++ b/docs/operators/spark-k8s-operator.md @@ -1,2 +1,336 @@ +--- +title: Spark +--- -# spark-k8s-operator +> Kubedoop Operator for Apache Spark — deploys and manages the Spark History Server on Kubernetes. + +## Overview + +[Apache Spark](https://spark.apache.org/) is a distributed engine for large-scale data processing. +When a Spark application runs it writes an *event log* describing every job, stage and task it +executed. Once the application finishes, its driver UI disappears along with it, and that event log +is all that remains. + +The **Spark History Server** reads those event logs back and serves the same web UI for applications +that have already completed. It is how you answer "why was last night's job slow?" after the job is +gone. + +This Operator manages the History Server. It does **not** submit or schedule Spark applications — +there is no `SparkApplication` resource in this API. You run Spark however you already do, point it +at an object store for event logs, and point a `SparkHistoryServer` at the same location to browse +them. + +## Prerequisites + +- Kubernetes 1.29+ +- kubectl +- Helm v3+ — see [Installation](../quick-start/installation.md) +- An S3-compatible object store holding the event logs, reachable from the cluster + +## Quick Start + +### Install the Operator + +Install the built-in Operators first — they are required: + +```bash +helm install commons-operator kubedoop/commons-operator -n operators --create-namespace +helm install listener-operator kubedoop/listener-operator -n operators +helm install secret-operator kubedoop/secret-operator -n operators +``` + +Then the Spark Operator: + +```bash +helm install spark-k8s-operator kubedoop/spark-k8s-operator -n operators +``` + +Verify it is running: + +```bash +kubectl get pods -n operators -l app.kubernetes.io/name=spark-k8s-operator +``` + +### Create a Namespace + +```bash +kubectl create ns spark +``` + +### Point at the event logs + +The History Server reads event logs from S3, so describe the bucket first. Credentials come from a +SecretClass, the connection from an `S3Connection`, and the bucket from an `S3Bucket`: + +```yaml +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Connection +metadata: + name: minio + namespace: spark +spec: + host: minio + port: 9000 + credentials: + secretClass: s3-credentials +--- +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Bucket +metadata: + name: spark-history + namespace: spark +spec: + bucketName: spark-history + connection: + reference: minio +``` + +> For the full set of connection options — TLS, region, and the `pathStyle` setting that most +> self-hosted backends need — see [S3](../core-concepts/resources/s3.md). + +### Deploy the History Server + +```bash +kubectl apply -f - < For more details, see +> [Roles and Role Groups](../core-concepts/common-configuration-mechanisms/roles-and-role-groups.md). + +### Configurations + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `clusterConfig.logFileDirectory.s3.bucket` | The bucket holding event logs, `inline` or `reference` | required | +| `clusterConfig.logFileDirectory.s3.prefix` | Path within the bucket | required | +| `clusterConfig.listenerClass` | How the UI is exposed | `cluster-internal` | +| `clusterConfig.authentication` | OIDC authentication for the UI | none | +| `clusterConfig.vectorAggregatorConfigMapName` | Vector aggregator discovery ConfigMap | none | +| `image.productVersion` | Spark version | `3.5.5` | +| `image.repo` | Image repository | `quay.io/zncdatadev` | +| `image.pullPolicy` | Image pull policy | `IfNotPresent` | +| `node.roleGroups..replicas` | Replicas in the group | `1` | +| `node.config.cleaner` | Enable event log cleanup | `false` | + +`clusterConfig` and `node` are both required; a resource without them is rejected. + +> Configuration can be set at the role level and overridden per role group — see +> [Overrides](../core-concepts/common-configuration-mechanisms/overrides.md). + +### Ports + +| Name | Port | Purpose | +|------|------|---------| +| `http` | 18080 | History Server web UI | +| `metrics` | 18081 | Metrics endpoint | +| `oidc` | 4180 | OIDC proxy, only when authentication is configured | + +### Listeners and Services + +`clusterConfig.listenerClass` decides how the UI is reachable: + +| Value | Exposure | +|-------|----------| +| `cluster-internal` (default) | ClusterIP — inside the cluster only | +| `external-unstable` | NodePort — address changes if the pod moves | +| `external-stable` | LoadBalancer — stable address | + +> For more details, see [Service Discovery](../core-concepts/connectivity/service-discovery.md). + +### Dependencies + +| Dependency | Required | Description | +|------------|----------|-------------| +| S3-compatible storage | Yes | Where event logs are read from; there is no other backend | +| `commons-operator`, `listener-operator`, `secret-operator` | Yes | Provide pod enrichment, service exposure and credential delivery | +| An OIDC provider | No | Only when `clusterConfig.authentication` is set | + +## Advanced + +### The event log cleaner + +Setting `cleaner: true` lets the History Server delete old event logs from the bucket. Because the +bucket is shared, **only one instance may clean**, and the Operator enforces that: + +```yaml +spec: + node: + roleGroups: + default: + replicas: 1 + config: + cleaner: true +``` + +Two configurations are rejected with an error rather than silently allowing several cleaners to race: + +- `cleaner` enabled on a role group whose `replicas` is greater than 1 +- `cleaner` enabled at the role level when the role has more than one role group + +If you need several role groups, enable the cleaner on exactly one single-replica group. + +### Resource Management + +```yaml +spec: + node: + config: + resources: + cpu: + min: "1" + max: "2" + memory: + limit: "4Gi" +``` + +> For more details, see [Resource Management](../core-concepts/resources/resource-manage.md). + +### Pod Placement + +Affinity is set under `config.affinity` on the role or a role group. Note that it **replaces** the +product's defaults rather than merging with them. + +> For more details, see [Pod Placement](../core-concepts/operations/pod-placement.md). + +### Authentication + +The UI can be put behind an OIDC provider by referencing an `AuthenticationClass`: + +```yaml +spec: + clusterConfig: + authentication: + authenticationClass: keycloak + oidc: + clientCredentialsSecret: spark-oidc-client + extraScopes: + - profile +``` + +The referenced Secret must contain `CLIENT_ID` and `CLIENT_SECRET`. When authentication is +configured the OIDC proxy port (4180) is added to the pod. + +> For more details, see [Authentication](../core-concepts/security/authentication.md). + +### Logging + +```yaml +spec: + node: + config: + logging: + containers: + node: + console: + level: INFO + file: + level: DEBUG +``` + +> For more details, see [Logging](../core-concepts/observability/logging.md). + +## Troubleshooting + +### Common Issues + +1. **The UI loads but lists no applications** + - **Symptom**: The History Server is running and reachable, but the application list is empty. + - **Cause**: `prefix` does not match the path your Spark applications write to, or the + applications are not writing event logs at all. + - **Resolution**: Confirm `spark.eventLog.enabled` is true in your applications and that + `spark.eventLog.dir` resolves to the same bucket and prefix. List the bucket to confirm files + are actually arriving. + +2. **The pod fails to start or cannot read the bucket** + - **Symptom**: The pod crash-loops, or the logs show S3 access errors. + - **Cause**: Wrong credentials, or virtual-host addressing against a backend that only serves + path-style. Nothing rejects the latter at apply time — it fails on first access. + - **Resolution**: Check that the SecretClass supplies `ACCESS_KEY` and `SECRET_KEY`, and set + `pathStyle: true` on the `S3Connection` for MinIO, Ceph RGW and similar backends. See + [S3](../core-concepts/resources/s3.md). + +3. **The resource is rejected when enabling the cleaner** + - **Symptom**: An error mentioning more than one role group or more than one replica with the + cleaner enabled. + - **Cause**: Several instances would compete to delete from the same bucket. + - **Resolution**: Enable `cleaner` on exactly one role group with `replicas: 1`. + +> For general operational topics, see +> [Pod Disruptions](../core-concepts/operations/pod-disruptions.md). + +## Clean Up + +```bash +kubectl delete sparkhistoryserver simple-history -n spark +kubectl delete ns spark +helm uninstall spark-k8s-operator -n operators +``` + +## Related Links + +- [Spark History Server documentation](https://spark.apache.org/docs/latest/monitoring.html) +- [Kubedoop Operator for Apache Spark on GitHub](https://github.com/zncdatadev/spark-k8s-operator) +- [Apache Spark on GitHub](https://github.com/apache/spark) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/operators/spark-k8s-operator.md b/i18n/zh/docusaurus-plugin-content-docs/current/operators/spark-k8s-operator.md index f0ee86fa..84905982 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/operators/spark-k8s-operator.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/operators/spark-k8s-operator.md @@ -1,4 +1,326 @@ +--- +title: Spark +--- -# spark-k8s-operator +> Kubedoop Operator for Apache Spark —— 在 Kubernetes 上部署和管理 Spark History Server。 -TODO +## 概述 + +[Apache Spark](https://spark.apache.org/) 是用于大规模数据处理的分布式引擎。Spark 应用运行时会写出 +**事件日志**(event log),记录它执行过的每个 job、stage 和 task。应用一旦结束,driver UI 也随之消失, +留下的就只有这份事件日志。 + +**Spark History Server** 负责把这些事件日志读回来,为已经结束的应用重新提供同样的 Web UI。 +当你想回答"昨晚那个任务为什么慢"时,靠的就是它。 + +本 Operator 管理的是 History Server。它**不负责**提交或调度 Spark 应用——该 API 中并没有 +`SparkApplication` 资源。你照旧用自己的方式运行 Spark,让它把事件日志写到对象存储, +再让 `SparkHistoryServer` 指向同一位置来浏览这些日志。 + +## 前置条件 + +- Kubernetes 1.29+ +- kubectl +- Helm v3+ —— 参见[安装](../quick-start/installation.md) +- 一个集群内可访问、存放事件日志的 S3 兼容对象存储 + +## 快速开始 + +### 安装 Operator + +先安装内置 Operator,它们是必需的: + +```bash +helm install commons-operator kubedoop/commons-operator -n operators --create-namespace +helm install listener-operator kubedoop/listener-operator -n operators +helm install secret-operator kubedoop/secret-operator -n operators +``` + +然后安装 Spark Operator: + +```bash +helm install spark-k8s-operator kubedoop/spark-k8s-operator -n operators +``` + +确认它已运行: + +```bash +kubectl get pods -n operators -l app.kubernetes.io/name=spark-k8s-operator +``` + +### 创建命名空间 + +```bash +kubectl create ns spark +``` + +### 指向事件日志 + +History Server 从 S3 读取事件日志,所以先把桶描述出来。凭据来自 SecretClass,连接来自 +`S3Connection`,桶来自 `S3Bucket`: + +```yaml +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Connection +metadata: + name: minio + namespace: spark +spec: + host: minio + port: 9000 + credentials: + secretClass: s3-credentials +--- +apiVersion: s3.kubedoop.dev/v1alpha1 +kind: S3Bucket +metadata: + name: spark-history + namespace: spark +spec: + bucketName: spark-history + connection: + reference: minio +``` + +> 完整的连接选项——TLS、region,以及大多数自建后端都需要的 `pathStyle`——参见 +> [S3](../core-concepts/resources/s3.md)。 + +### 部署 History Server + +```bash +kubectl apply -f - < 更多内容参见 +> [角色和角色组](../core-concepts/common-configuration-mechanisms/roles-and-role-groups.md)。 + +### 配置项 + +| 参数 | 说明 | 默认值 | +|------|------|--------| +| `clusterConfig.logFileDirectory.s3.bucket` | 存放事件日志的桶,`inline` 或 `reference` | 必填 | +| `clusterConfig.logFileDirectory.s3.prefix` | 桶内路径 | 必填 | +| `clusterConfig.listenerClass` | UI 的暴露方式 | `cluster-internal` | +| `clusterConfig.authentication` | UI 的 OIDC 认证 | 无 | +| `clusterConfig.vectorAggregatorConfigMapName` | Vector 聚合端发现用 ConfigMap | 无 | +| `image.productVersion` | Spark 版本 | `3.5.5` | +| `image.repo` | 镜像仓库 | `quay.io/zncdatadev` | +| `image.pullPolicy` | 镜像拉取策略 | `IfNotPresent` | +| `node.roleGroups..replicas` | 该组副本数 | `1` | +| `node.config.cleaner` | 是否启用事件日志清理 | `false` | + +`clusterConfig` 和 `node` 都是必填的,缺少任一的资源会被拒绝。 + +> 配置可以在角色级设置、并在角色组级覆盖——参见 +> [覆盖配置](../core-concepts/common-configuration-mechanisms/overrides.md)。 + +### 端口 + +| 名称 | 端口 | 用途 | +|------|------|------| +| `http` | 18080 | History Server Web UI | +| `metrics` | 18081 | 指标端点 | +| `oidc` | 4180 | OIDC 代理,仅在配置了认证时存在 | + +### 监听器与服务 + +`clusterConfig.listenerClass` 决定 UI 如何被访问: + +| 取值 | 暴露方式 | +|------|----------| +| `cluster-internal`(默认) | ClusterIP——仅集群内可访问 | +| `external-unstable` | NodePort——Pod 漂移后地址会变 | +| `external-stable` | LoadBalancer——地址稳定 | + +> 更多内容参见[服务发现](../core-concepts/connectivity/service-discovery.md)。 + +### 依赖 + +| 依赖 | 必需 | 说明 | +|------|------|------| +| S3 兼容存储 | 是 | 事件日志的唯一来源,没有其他后端可选 | +| `commons-operator`、`listener-operator`、`secret-operator` | 是 | 分别提供 Pod 信息补全、服务暴露和凭据分发 | +| OIDC 提供方 | 否 | 仅在设置了 `clusterConfig.authentication` 时需要 | + +## 进阶 + +### 事件日志清理器 + +设置 `cleaner: true` 可以让 History Server 删除桶中的旧事件日志。由于这个桶是共享的, +**只能有一个实例执行清理**,Operator 会强制这一点: + +```yaml +spec: + node: + roleGroups: + default: + replicas: 1 + config: + cleaner: true +``` + +以下两种配置会直接报错,而不是放任多个清理器互相竞争: + +- 在 `replicas` 大于 1 的角色组上启用 `cleaner` +- 角色拥有多个角色组时,在角色级启用 `cleaner` + +如果确实需要多个角色组,请只在其中一个单副本组上启用清理器。 + +### 资源管理 + +```yaml +spec: + node: + config: + resources: + cpu: + min: "1" + max: "2" + memory: + limit: "4Gi" +``` + +> 更多内容参见[资源管理](../core-concepts/resources/resource-manage.md)。 + +### Pod 放置 + +亲和性配置在角色或角色组的 `config.affinity` 下。注意它是**替换**产品默认值,而非与之合并。 + +> 更多内容参见[Pod 放置](../core-concepts/operations/pod-placement.md)。 + +### 认证 + +可以通过引用 `AuthenticationClass` 把 UI 置于 OIDC 提供方之后: + +```yaml +spec: + clusterConfig: + authentication: + authenticationClass: keycloak + oidc: + clientCredentialsSecret: spark-oidc-client + extraScopes: + - profile +``` + +被引用的 Secret 必须包含 `CLIENT_ID` 和 `CLIENT_SECRET`。配置认证后,Pod 上会增加 OIDC 代理端口 +(4180)。 + +> 更多内容参见[认证](../core-concepts/security/authentication.md)。 + +### 日志 + +```yaml +spec: + node: + config: + logging: + containers: + node: + console: + level: INFO + file: + level: DEBUG +``` + +> 更多内容参见[日志](../core-concepts/observability/logging.md)。 + +## 故障排查 + +### 常见问题 + +1. **UI 能打开,但没有任何应用** + - **现象**:History Server 正常运行也能访问,但应用列表是空的。 + - **原因**:`prefix` 与 Spark 应用实际写入的路径不一致,或者应用根本没有写事件日志。 + - **解决**:确认应用的 `spark.eventLog.enabled` 为 true,且 `spark.eventLog.dir` + 指向同一个桶和前缀。直接列一下桶,确认文件确实在写进来。 + +2. **Pod 启动失败或读不到桶** + - **现象**:Pod 反复重启,或日志里出现 S3 访问错误。 + - **原因**:凭据不对;或者对只支持路径风格的后端使用了虚拟主机寻址。后者在 apply 阶段不会被拦截, + 只会在首次访问时失败。 + - **解决**:确认 SecretClass 提供了 `ACCESS_KEY` 和 `SECRET_KEY`;对 MinIO、Ceph RGW + 这类后端,在 `S3Connection` 上设置 `pathStyle: true`。参见 + [S3](../core-concepts/resources/s3.md)。 + +3. **启用 cleaner 时资源被拒绝** + - **现象**:报错提示启用了 cleaner 的角色组数量或副本数大于 1。 + - **原因**:多个实例会争抢删除同一个桶中的内容。 + - **解决**:只在一个 `replicas: 1` 的角色组上启用 `cleaner`。 + +> 通用运维话题参见 [Pod 干扰](../core-concepts/operations/pod-disruptions.md)。 + +## 清理 + +```bash +kubectl delete sparkhistoryserver simple-history -n spark +kubectl delete ns spark +helm uninstall spark-k8s-operator -n operators +``` + +## 相关链接 + +- [Spark History Server 官方文档](https://spark.apache.org/docs/latest/monitoring.html) +- [Kubedoop Operator for Apache Spark(GitHub)](https://github.com/zncdatadev/spark-k8s-operator) +- [Apache Spark(GitHub)](https://github.com/apache/spark)