docs(operators): write the Spark operator page - #42
Merged
Conversation
First real page under operators/, filled from _template.md's section order. Content comes from the spark-k8s-operator repository: the CRD in api/v1alpha1, the port constants in internal/util, the cleaner constraints in internal/controller/historyserver/configmap.go, and a working manifest from test/e2e/smoke/submit-task. Leads with what the operator actually is. The repository name suggests Spark on Kubernetes generally, but the API has exactly one kind, SparkHistoryServer — there is no SparkApplication resource, and the operator does not submit or schedule jobs. Saying so up front is the single most useful thing this page does. Documents the single `node` role, the required clusterConfig.logFileDirectory.s3 bucket and prefix, listenerClass and what each value exposes, the three ports, OIDC authentication, and defaults taken from the CRD rather than guessed. Gives the event log cleaner its own section: it deletes from shared object storage, so the operator rejects `cleaner` on a role group with more than one replica, and rejects it at role level when the role has more than one role group. Note for future pages: MDX 3 parses `<http://...>` as a JSX tag and fails the build. Use [text](url) form for bare URLs in docs pages. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 tasks
whg517
added a commit
that referenced
this pull request
Aug 27, 2026
… a version claim (#43) * docs(user-manual): write the Kubernetes environment page Replaces a one-line placeholder. The supported-version table comes from the chainsaw job's matrix in each operator's test.yml and release.yml, which is consistent across repositories: 1.33.7, 1.34.3, 1.35.0. The cluster requirements come from what the built-in operators actually install. secret-operator and listener-operator are CSI drivers, not ordinary controllers — they register CSIDriver objects and run node DaemonSets that mount the kubelet directory, so a cluster that forbids that cannot run them, and without them no product cluster gets secrets or service exposure. Calls out the kubelet directory explicitly. The chart defaults kubeletDir to /var/lib/kubelet and its own values.yaml notes the path varies on microk8s; when it is wrong the driver never registers and every pod wanting a secret volume sits in Pending with nothing obviously broken. Includes the command to find the real path. Also notes that the charts declare no kubeVersion constraint, so the version table is a statement of what is tested rather than something the tooling enforces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: correct the Kubernetes version claim in two pages #41 and #42 both stated a 1.29 floor. That number came from a comment next to KIND_K8S_VERSION in zookeeper-operator's Makefile, which is the default for a locally created kind cluster — not the project's tested range, and not even consistent across repositories: zookeeper-operator defaults to 1.36.1 while hdfs, trino, secret, listener and commons all default to 1.26.15. What CI actually exercises is a matrix in the chainsaw job, identical in every operator: 1.33.7, 1.34.3 and 1.35.0. The workspace-level notes say 1.26+, which is a third answer again. Point both pages at the new Kubernetes page instead of restating a number, and note in the development guideline that KIND_K8S_VERSION is a local default CI overrides, so the next reader does not draw the same wrong conclusion from it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First real page under
operators/— it was a two-line stub. Filled following_template.md's section order, in both language trees.Content comes from the
spark-k8s-operatorrepository: the CRD inapi/v1alpha1, the port constants ininternal/util, the cleaner constraints ininternal/controller/historyserver/configmap.go, and a working manifest fromtest/e2e/smoke/submit-task.The most useful thing this page does
It says what the operator actually is. The repository name suggests Spark on Kubernetes generally, but the API has exactly one kind —
SparkHistoryServer. There is noSparkApplicationresource; the operator does not submit or schedule jobs. It manages the History Server, which reads event logs your Spark applications already write to S3.Someone installing this expecting a job submitter would otherwise find that out the hard way.
What it covers
noderoleclusterConfig.logFileDirectory.s3bucket + prefix, with the full S3 setup chain (SecretClass→S3Connection→S3Bucket) taken from the e2e setup manifestslistenerClassand what each of the three values exposeshttp18080,metrics18081,oidc4180AuthenticationClass, and theCLIENT_ID/CLIENT_SECRETcontractproductVersion: 3.5.5,repo: quay.io/zncdatadev,listenerClass: cluster-internal,replicas: 1) rather than guessedThe event log cleaner gets its own section
cleaner: truedeletes old event logs from shared object storage, so only one instance may do it. The operator enforces this and rejects two configurations rather than letting cleaners race:cleaneron a role group withreplicas > 1cleanerat role level when the role has more than one role groupTesting
npm run verifyexits 0 — 52 files linted with 0 errors, both locales builtonBrokenLinks: 'throw') and no anchor warnings in the build log<cluster>-<role>-<group>)One thing worth knowing for future pages
MDX 3 parses
<http://...>as a JSX tag and fails the build:Autolink syntax cannot be used for URLs in doc pages — use
[text](url)instead. Hit this on the first build and fixed it here; noted in the commit body so the next person does not rediscover it.Remaining stage 2 placeholders
5 left:
kubernetes,kubebuilder,faq,release-notes, anddatabase(blocked, see #38).faqandrelease-notesstill need maintainer input rather than source research.🤖 Generated with Claude Code