Skip to content

docs(operators): write the Spark operator page - #42

Merged
whg517 merged 1 commit into
zncdatadev:mainfrom
whg517:docs/spark-history-server
Aug 25, 2026
Merged

docs(operators): write the Spark operator page#42
whg517 merged 1 commit into
zncdatadev:mainfrom
whg517:docs/spark-history-server

Conversation

@whg517

@whg517 whg517 commented Aug 24, 2026

Copy link
Copy Markdown
Member

Summary

First real page under operators/ — it was a two-line stub. Filled following _template.md's section order, in both language trees.

Content comes from the spark-k8s-operator repository: the CRD in api/v1alpha1, the port constants in internal/util, the cleaner constraints in internal/controller/historyserver/configmap.go, and a working manifest from test/e2e/smoke/submit-task.

The most useful thing this page does

It says what the operator actually is. The repository name suggests Spark on Kubernetes generally, but the API has exactly one kind — SparkHistoryServer. There is no SparkApplication resource; the operator does not submit or schedule jobs. It manages the History Server, which reads event logs your Spark applications already write to S3.

Someone installing this expecting a job submitter would otherwise find that out the hard way.

What it covers

  • The single node role
  • Required clusterConfig.logFileDirectory.s3 bucket + prefix, with the full S3 setup chain (SecretClassS3ConnectionS3Bucket) taken from the e2e setup manifests
  • listenerClass and what each of the three values exposes
  • Ports: http 18080, metrics 18081, oidc 4180
  • OIDC authentication via AuthenticationClass, and the CLIENT_ID/CLIENT_SECRET contract
  • Defaults read from the CRD (productVersion: 3.5.5, repo: quay.io/zncdatadev, listenerClass: cluster-internal, replicas: 1) rather than guessed

The event log cleaner gets its own section

cleaner: true deletes old event logs from shared object storage, so only one instance may do it. The operator enforces this and rejects two configurations rather than letting cleaners race:

  • cleaner on a role group with replicas > 1
  • cleaner at role level when the role has more than one role group

Testing

  • npm run verify exits 0 — 52 files linted with 0 errors, both locales built
  • Page renders in both locales
  • All relative links resolve (onBrokenLinks: 'throw') and no anchor warnings in the build log
  • Resource naming in the examples checked against the e2e assert (<cluster>-<role>-<group>)

One thing worth knowing for future pages

MDX 3 parses <http://...> as a JSX tag and fails the build:

Unexpected character `/` (U+002F) before local name, expected a character that can start a name

Autolink syntax cannot be used for URLs in doc pages — use [text](url) instead. Hit this on the first build and fixed it here; noted in the commit body so the next person does not rediscover it.

Remaining stage 2 placeholders

5 left: kubernetes, kubebuilder, faq, release-notes, and database (blocked, see #38). faq and release-notes still need maintainer input rather than source research.

🤖 Generated with Claude Code

First real page under operators/, filled from _template.md's section
order. Content comes from the spark-k8s-operator repository: the CRD in
api/v1alpha1, the port constants in internal/util, the cleaner
constraints in internal/controller/historyserver/configmap.go, and a
working manifest from test/e2e/smoke/submit-task.

Leads with what the operator actually is. The repository name suggests
Spark on Kubernetes generally, but the API has exactly one kind,
SparkHistoryServer — there is no SparkApplication resource, and the
operator does not submit or schedule jobs. Saying so up front is the
single most useful thing this page does.

Documents the single `node` role, the required
clusterConfig.logFileDirectory.s3 bucket and prefix, listenerClass and
what each value exposes, the three ports, OIDC authentication, and
defaults taken from the CRD rather than guessed.

Gives the event log cleaner its own section: it deletes from shared
object storage, so the operator rejects `cleaner` on a role group with
more than one replica, and rejects it at role level when the role has
more than one role group.

Note for future pages: MDX 3 parses `<http://...>` as a JSX tag and
fails the build. Use [text](url) form for bare URLs in docs pages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@whg517
whg517 merged commit 27d7b9c into zncdatadev:main Aug 25, 2026
3 checks passed
whg517 added a commit that referenced this pull request Aug 27, 2026
… a version claim (#43)

* docs(user-manual): write the Kubernetes environment page

Replaces a one-line placeholder. The supported-version table comes from
the chainsaw job's matrix in each operator's test.yml and release.yml,
which is consistent across repositories: 1.33.7, 1.34.3, 1.35.0.

The cluster requirements come from what the built-in operators actually
install. secret-operator and listener-operator are CSI drivers, not
ordinary controllers — they register CSIDriver objects and run node
DaemonSets that mount the kubelet directory, so a cluster that forbids
that cannot run them, and without them no product cluster gets secrets
or service exposure.

Calls out the kubelet directory explicitly. The chart defaults
kubeletDir to /var/lib/kubelet and its own values.yaml notes the path
varies on microk8s; when it is wrong the driver never registers and every
pod wanting a secret volume sits in Pending with nothing obviously
broken. Includes the command to find the real path.

Also notes that the charts declare no kubeVersion constraint, so the
version table is a statement of what is tested rather than something the
tooling enforces.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: correct the Kubernetes version claim in two pages

#41 and #42 both stated a 1.29 floor. That number came from a comment
next to KIND_K8S_VERSION in zookeeper-operator's Makefile, which is the
default for a locally created kind cluster — not the project's tested
range, and not even consistent across repositories: zookeeper-operator
defaults to 1.36.1 while hdfs, trino, secret, listener and commons all
default to 1.26.15.

What CI actually exercises is a matrix in the chainsaw job, identical in
every operator: 1.33.7, 1.34.3 and 1.35.0. The workspace-level notes say
1.26+, which is a third answer again.

Point both pages at the new Kubernetes page instead of restating a
number, and note in the development guideline that KIND_K8S_VERSION is a
local default CI overrides, so the next reader does not draw the same
wrong conclusion from it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant