feat(operator): report whether the Secret/ConfigMap write policy is in force (#2500)

.changeset/operator-write-policy-selfcheck.md

@@ -0,0 +1,5 @@
+---
+"@getknext/core": patch
+---
+
+`knext doctor` now checks that the operator's Secret and ConfigMap write policy is in force. The policy, a `ValidatingAdmissionPolicy` shipped in the install bundle, is the only thing that limits the operator to its per-zone signing-key Secrets and JWKS ConfigMaps. It can be off with nothing saying so: a cluster older than Kubernetes 1.30 rejects it while the rest of the bundle installs, an install under a different namespace or ServiceAccount name than the bundle rendered leaves a policy that names a user the operator is not, and an admin or a GitOps prune can delete it. The new `write-policy` row reports a `WARN` that names which of these it is, and `PASS` only when the policy exists, covers create/update/delete on Secrets and ConfigMaps without narrowing, carries the validations this release ships, has a binding that denies, and names the ServiceAccount the operator runs as. A read it cannot classify or parse is a `WARN`, never a skip. The operator reports the same verdict as the `knext_operator_write_policy_enforced` gauge (re-checked every five minutes, warning log on change, still starts when it is `0`), with a `KnextOperatorWritePolicyNotEnforced` alert (a `0`) and a `KnextOperatorWritePolicyUnreported` alert (an operator pod that is scraped but exports no gauge). The operator's role gains `get` on exactly the policy and its binding, by name, and nothing else.

apps/docs/content/docs/cli.mdx

@@ -478,6 +478,19 @@ whether your cluster's CNI can **enforce** the default-on `NetworkPolicy` the op
 on flannel it cannot (the policy is declarative only), and doctor says so plainly rather than
 guessing; "cannot determine" is reported as its own outcome, never as "enforced".
 
+**Operator write policy.** The operator's Secret and ConfigMap writes are limited by a
+`ValidatingAdmissionPolicy` that can be off with nothing failing: a server older than Kubernetes
+1.30, an install under a different namespace or ServiceAccount name than the bundle rendered, or a
+deleted policy or binding. Doctor reads the policy and its binding by name and reports `PASS` only
+when the policy exists, covers create, update and delete on Secrets and ConfigMaps without narrowing
+(no `resourceNames`, `Cluster` scope or `excludeResourceRules`), carries the validations this release
+ships, has a binding that includes `Deny`, and names the ServiceAccount the operator Deployment runs
+as. Anything else, including output it cannot read, is a `WARN` that says which part is wrong, and a server
+older than 1.30 is a `WARN` naming the floor. A read that RBAC denies is a `WARN` ("cannot
+determine"), never a pass. It never fails the run, because the missing safety net is not an
+outage; the operator publishes the same verdict as the `knext_operator_write_policy_enforced` gauge
+(see [Security](/docs/security#write-policy-enforced)).
+
 **Redis key prefixes.** When any `NextApp` uses a Redis cache, doctor compares the apps' key
 prefixes: `cache.keyPrefix`, a `REDIS_KEY_PREFIX` in `env`, or the per-app default the operator
 derives. It `WARN`s, naming the apps, when:

apps/docs/content/docs/security.mdx

@@ -474,6 +474,56 @@ Two limits. The policy checks name and type, not ownership: a compromised operat
 change an `Opaque` Secret or a ConfigMap that happens to carry the name of a zone's key or key set.
 And a cluster administrator can remove the policy.
 
+#### Knowing whether the policy is in force [#write-policy-enforced]
+
+The policy can be off without anything failing. A cluster older than Kubernetes 1.30 rejects the two
+objects while the rest of the bundle installs. An install under a different namespace or
+ServiceAccount name than the bundle rendered leaves a policy that names a user the operator is not.
+An administrator, or a GitOps tool pruning objects it does not own, can delete either object. In
+each case the operator is simply unrestricted.
+
+So the operator checks for itself, when it starts and every five minutes after. It reads the policy
+and its binding (a read-only `get` on those two names, nothing else) and compares the policy's
+username with the one the API server reports for the operator. It publishes the result as a gauge:
+
+| Metric | Meaning |
+|---|---|
+| `knext_operator_write_policy_enforced` | `1` when the policy exists, its binding denies, and the policy names the identity the operator runs as. `0` when any of that is false, or when the objects could not be read. |
+
+It logs one warning each time the answer changes to "not in force", naming the reason. The operator
+still starts: refusing to run is your decision, not the operator's. The `KnextOperatorWritePolicyNotEnforced`
+alert (warning, 15 minutes) fires on a `0`; see [SLOs & alerting](/docs/slos-alerting).
+
+The gauge reads `0` for any of these:
+
+- the policy or the binding is missing;
+- the binding does not include `Deny` (a binding that only audits or warns refuses nothing), or it
+  limits the policy to some namespaces;
+- the policy names a different user than the operator runs as;
+- the policy fails open (`failurePolicy: Ignore`);
+- the policy does not cover create, update and delete on both Secrets and ConfigMaps in the core
+  `v1` API: a rule that names objects (`resourceNames`), is scoped `Cluster`, or is for another API
+  version does not count, a namespace or object selector narrows it, and any `excludeResourceRules`
+  entry is treated as removing something;
+- the policy has a condition the check cannot read as an exact user;
+- the policy declares parameters (`paramKind`) or its binding sets a `paramRef`. The shipped policy
+  takes none, and a parameter that cannot be found makes the API server skip every validation;
+- the policy's validations are not the ones this release ships. The check compares the policy's
+  variables (the allow-lists and the Secret type) and each validation's expression and
+  `messageExpression` with the shipped copy, so a validation replaced by `true` is caught. The
+  `message` text and the `reason` are not compared. An operator newer than the policy installed in the
+  cluster reads `0` here until you reapply the bundle.
+
+Everything the check cannot read, it counts as not in force.
+
+If the operator exports no such gauge at all (a release older than the check, or a failing metrics
+scrape), the `0` alert cannot fire, so a second alert, `KnextOperatorWritePolicyUnreported`, fires
+for an operator pod that Prometheus has discovered but that exports no gauge.
+
+`knext doctor` reports the same verdict from your machine, plus the server version against the 1.30
+floor. A read it cannot classify or parse is a `WARN`, not a skip. It reads the policy name from the operator's `WRITE_POLICY_NAME` environment variable, which
+the bundle sets, so a renamed install is checked under its own name.
+
 The policy needs **Kubernetes 1.30 or later**, where `ValidatingAdmissionPolicy` became generally
 available. That is the same server floor as the rest of knext (see the
 [support matrix](/docs/support-matrix#kubernetes)). On an older cluster the API server does not

apps/docs/content/docs/slos-alerting.mdx

@@ -20,7 +20,7 @@ catalog and scrape setup see [Observability](/docs/observability); for logs see
 | `knext_bunexec_http_inflight_requests` | App runtime. Saturation |
 | `knext_bunexec_startup_duration_seconds` | App runtime. Set once per process, when the listener binds — the cold-start signal |
 | `knext_bunexec_process_uptime_seconds`, `knext_bunexec_process_resident_memory_bytes` | App runtime |
-| `knext_nextapp_reconcile_total{result}`, `knext_nextapp_reconcile_errors_total`, `knext_nextapp_reconcile_duration_seconds_bucket`, `knext_nextapp_image_prewarm_errors_total` | Operator |
+| `knext_nextapp_reconcile_total{result}`, `knext_nextapp_reconcile_errors_total`, `knext_nextapp_reconcile_duration_seconds_bucket`, `knext_nextapp_image_prewarm_errors_total`, `knext_operator_write_policy_enforced` | Operator |
 | `workqueue_depth{name="nextapp"}` | Operator (control-plane saturation) |
 | `knext_nextapp_condition{type,status,namespace,name}` | kube-state-metrics, reading `NextApp.status.conditions` ([setup below](#kube-state-metrics)) |
 | `up{job="knext-nextapp"}` | Prometheus itself — scrape health, and the anchor for the meta-alerts |
@@ -122,6 +122,39 @@ histogram_quantile(0.95,
 Alert `KnextOperatorReconcileSlow` (p95 above 30 seconds for 15 minutes, warning). A related alert,
 `KnextImagePrewarmFailing` (warning), fires when image pre-warming errors persist for 15 minutes.
 
+## Operator write policy (control plane)
+
+**Objective:** the policy that limits the operator's Secret and ConfigMap writes is in force on every
+cluster that supports it.
+
+```text
+knext_operator_write_policy_enforced
+```
+
+The operator sets this gauge to `1` only when all of this holds, and to `0` otherwise, including when
+the objects cannot be read:
+
+- the policy exists and does not fail open;
+- it covers create, update and delete on Secrets and ConfigMaps (core `v1`), with no `resourceNames`,
+  no `Cluster` scope, no `excludeResourceRules` and no namespace or object selector;
+- its condition is an exact match on the user the operator runs as;
+- its variables and validations (their expressions and `messageExpression`) are the ones the
+  operator release ships;
+- neither the policy declares `paramKind` nor its binding sets a `paramRef` (a parameter that cannot
+  be found makes the API server skip every validation);
+- its binding exists, names the policy, includes `Deny` and is not limited to some namespaces.
+
+Alert `KnextOperatorWritePolicyNotEnforced` (a `0` for 15 minutes, warning, one series per operator
+pod). It is a warning because a missing safety net is not an outage, and a cluster older than
+Kubernetes 1.30 cannot hold the policy at all.
+
+An alert on `== 0` says nothing when the series is missing, so a companion alert,
+`KnextOperatorWritePolicyUnreported` (warning, 15 minutes), fires for an operator pod that Prometheus
+scrapes (`up` exists) but that exports no such gauge: an operator older than the check, or a metrics
+scrape that is failing. It is keyed on a discovered target rather than a bare `absent()`, so it stays
+quiet on a cluster with no operator at all. `knext doctor` names the reason; see
+[Security](/docs/security#write-policy-enforced).
+
 ## NextApp health (control plane)
 
 **Objective:** no `NextApp` stays `Degraded=True`.

apps/file-manager/src/app/api/_metrics/prometheusrule.test.ts

@@ -199,6 +199,74 @@ describe('PrometheusRule manifest', () => {
     expect(prewarm?.annotations?.runbook_url ?? prewarm?.annotations?.runbook).toBeTruthy();
   });
 
+  // The operator's Secret/ConfigMap write policy can be off with nothing saying
+  // so: a server older than 1.30 rejects it, a repackaged install leaves a
+  // matchCondition that matches nobody, a prune deletes it. The operator reports
+  // it as a 0/1 gauge; this alert is what makes a 0 visible.
+  it('alerts when the operator write policy is not in force', () => {
+    const { docs } = loadRule();
+    const rule = docs.find((d) => d?.kind === 'PrometheusRule');
+    const alerts: (Rule & { for?: string })[] = rule.spec.groups.flatMap(
+      (g: { rules: Rule[] }) => g.rules,
+    );
+    const off = alerts.find((a) => a.alert === 'KnextOperatorWritePolicyNotEnforced');
+
+    expect(off, 'KnextOperatorWritePolicyNotEnforced alert must exist').toBeDefined();
+    // Per pod, not aggregated: every replica reports, and one that cannot see the
+    // policy is itself a finding.
+    expect(off?.expr).toMatch(/knext_operator_write_policy_enforced\s*==\s*0/);
+    // Warning, not critical: it is a missing safety net, not an outage, and every
+    // pre-1.30 cluster would page forever otherwise. A "for" longer than the
+    // operator's re-check interval keeps a restart from flapping it.
+    expect(off?.labels?.severity).toBe('warning');
+    expect(off?.for).toBeTruthy();
+    expect(off?.annotations?.runbook_url ?? off?.annotations?.runbook).toBeTruthy();
+  });
+
+  // The alert above reads `== 0`, and PromQL over a series nobody emits returns
+  // an empty vector: an operator that never publishes the gauge (an older
+  // operator over a newer rule file, the monitor not started, a scrape that
+  // fails) would make it silent forever. This companion fires on a DISCOVERED
+  // operator target that exports no gauge. It keys on `up`, never a bare
+  // absent(): with no operator running at all there is nothing to report on,
+  // and an always-on absent() would fire on every cluster before install.
+  it('alerts when a scraped operator pod exports no write-policy gauge', () => {
+    const { docs } = loadRule();
+    const rule = docs.find((d) => d?.kind === 'PrometheusRule');
+    const alerts: (Rule & { for?: string })[] = rule.spec.groups.flatMap(
+      (g: { rules: Rule[] }) => g.rules,
+    );
+    const blind = alerts.find((a) => a.alert === 'KnextOperatorWritePolicyUnreported');
+
+    expect(blind, 'KnextOperatorWritePolicyUnreported alert must exist').toBeDefined();
+    expect(blind?.expr).toMatch(/\bup\{/);
+    expect(blind?.expr).not.toMatch(/\babsent\(/);
+    expect(blind?.expr).toMatch(
+      /unless\s+on\s*\(\s*namespace\s*,\s*pod\s*\)\s*knext_operator_write_policy_enforced/,
+    );
+    expect(blind?.labels?.severity).toBe('warning');
+    expect(blind?.for).toBeTruthy();
+    expect(blind?.annotations?.runbook_url ?? blind?.annotations?.runbook).toBeTruthy();
+
+    // The job selector must actually match the operator's metrics Service as the
+    // bundle renders it (name prefix + service name), or the alert selects
+    // nothing and is silent in exactly the way it exists to prevent.
+    const op = join(__dirname, '../../../../../../packages/kn-next-operator/config');
+    const prefix = /^namePrefix:\s*(\S+)/m.exec(
+      readFileSync(join(op, 'default/kustomization.yaml'), 'utf8'),
+    )?.[1];
+    const svc = /^kind: Service[\s\S]*?^ {2}name:\s*(\S+)/m.exec(
+      readFileSync(join(op, 'default/metrics_service.yaml'), 'utf8'),
+    )?.[1];
+    expect(prefix, 'namePrefix must be readable').toBeTruthy();
+    expect(svc, 'the metrics Service name must be readable').toBeTruthy();
+    const job = /job=~"([^"]+)"/.exec(blind?.expr ?? '')?.[1];
+    expect(job, 'the alert must select the operator by job=~"..."').toBeTruthy();
+    expect(new RegExp(`^(?:${job})$`).test(`${prefix}${svc}`)).toBe(true);
+    // ...and must not select the app pods or anything else.
+    expect(new RegExp(`^(?:${job})$`).test('knext-nextapp')).toBe(false);
+  });
+
   // The reconcile-error page must stay CRITICAL — the decoupling narrowed what
   // reaches it, and if that alert were softened too, nothing would page on a
   // genuinely failing control loop.

docs/security/threat-model.md

@@ -340,6 +340,38 @@ to replace it, and a ServiceAccount with the operator's name in another namespac
   bundle is one artifact; there is no policy-free variant (jev: align the docs 1.00, ship a second
   bundle 0.00). On an older cluster the apply rejects the two policy objects and the rest of the
   bundle applies, leaving the operator unrestricted.
+- *Silent-off detection (#2500).* Three ways leave the operator unrestricted with nothing reporting
+  it: the version floor above, an install under another namespace or ServiceAccount name than the
+  bundle rendered (a Helm or OLM repackage, a hand-edited overlay), and deletion of the policy or its
+  binding by an admin or a GitOps prune. `internal/writepolicy` is the operator's read-only
+  self-check: at start-up and every five minutes it `get`s the policy and binding, asks the API
+  server who it is (`SelfSubjectReview`, which every authenticated user may create), and publishes
+  `knext_operator_write_policy_enforced`. It is 1 only if ALL of this holds: the policy exists with
+  `failurePolicy` not `Ignore`; its `matchConstraints` cover create/update/delete on core `v1`
+  Secrets and ConfigMaps through a rule with no `resourceNames`, no `Cluster` scope and the `v1` (or
+  `*`) apiVersion, with no `excludeResourceRules` (any is treated as carving something out) and no
+  namespace/object selector; its matchCondition is an exact equality on the operator's own username;
+  **its variables and validations are the shipped CEL** (round-1 review finding: matchConstraints,
+  identity and binding can all be right while a validation was replaced by `true`; compared are each
+  variable's name and expression and each validation's expression and `messageExpression`, against a
+  copy in `canonical.go` / `write-policy.ts` that `TestCanonicalPolicyCELEqualsTheShippedPolicy` and
+  its TS twin fail on if `config/admission` changes without it; `message` and `reason` are cosmetic
+  and not compared; an operator newer than the installed policy reports `PolicyModified` until the
+  bundle is reapplied); the policy declares no `paramKind` and the binding sets no `paramRef`
+  (closing-review finding: a `paramKind` plus a `paramRef` that finds nothing, with
+  `parameterNotFoundAction: Allow`, makes the apiserver run no validation while everything else
+  checks out; the shipped bundle sets neither, so either is `PolicyModified` / `BindingScoped`); and
+  the binding exists, names the policy, includes `Deny` and is not scoped.
+  Unreadable or unknown is 0, never 1. The `== 0` alert is blind if the series is missing, so
+  `KnextOperatorWritePolicyUnreported` fires on a discovered operator target (`up`) with no gauge. It logs one warning per change and **does not
+  refuse to start** (that is a founder decision, not made here). The alert
+  `KnextOperatorWritePolicyNotEnforced` and `knext doctor`'s `write-policy` row report the same
+  verdict; both are held to one shared table (`internal/writepolicy/testdata/verdict-cases.json`).
+  The RBAC cost is `get` on exactly the two named objects (`config/rbac/write_policy_reader_role.yaml`,
+  pinned by `TestWritePolicyReaderRoleIsExactlyGetOnTheTwoNamedObjects`), a separate role because a
+  marker's `resourceNames` is not rewritten by the bundle's name prefix and the bundle rewrites it
+  from the policy's real name. **Residual:** a compromised operator can still read the two objects
+  it already could not change, and the check detects a missing control, it does not restore it.
 
 **Defence in depth: the function mounts the JWKS only if the zone controls it (#2464).** Before it
 renders a function, the BackendService reconciler reads the metadata of `<zone>-knext-jwks`