Skip to content

KEDA PR gate test, with cluster - #2

Open
pvishwa3 wants to merge 1 commit into
main-clusterfrom
keda-pr-cluster
Open

pvishwa3 wants to merge 1 commit into
main-clusterfrom
keda-pr-cluster

Conversation

@pvishwa3

Copy link
Copy Markdown

Should fail with 9 KEDA findings from the cluster checks.

@github-actions

Copy link
Copy Markdown

Deployment risk: HIGH (score 64) — 52 findings

Severity Rule Resource Title
MEDIUM wrs-rs-unset Deployment/pr-web No resource requests set — unpredictable scheduling, BestEffort eviction priority
MEDIUM wrs-rs-unset Deployment/pr-auth No resource requests set — unpredictable scheduling, BestEffort eviction priority
MEDIUM wrs-rs-unset Deployment/pr-long No resource requests set — unpredictable scheduling, BestEffort eviction priority
MEDIUM wrs-spec-no-readiness Deployment/pr-web No readiness probe — traffic may hit unready pods
MEDIUM wrs-spec-no-readiness Deployment/pr-auth No readiness probe — traffic may hit unready pods
MEDIUM wrs-spec-no-readiness Deployment/pr-long No readiness probe — traffic may hit unready pods
LOW wrs-spec-no-liveness Deployment/pr-web No liveness probe — hung pods won't auto-restart
LOW wrs-spec-no-liveness Deployment/pr-auth No liveness probe — hung pods won't auto-restart
LOW wrs-spec-no-liveness Deployment/pr-long No liveness probe — hung pods won't auto-restart
LOW wrs-rs-nolimit Deployment/pr-web No CPU/memory limit — noisy-neighbour risk
LOW wrs-rs-nolimit Deployment/pr-auth No CPU/memory limit — noisy-neighbour risk
LOW wrs-rs-nolimit Deployment/pr-long No CPU/memory limit — noisy-neighbour risk
LOW mrs-runasroot-unspecified Deployment/pr-web runAsNonRoot not set — container may run as root
LOW mrs-runasroot-unspecified Deployment/pr-auth runAsNonRoot not set — container may run as root
LOW mrs-runasroot-unspecified Deployment/pr-long runAsNonRoot not set — container may run as root
LOW mrs-labels-missing-required Deployment/pr-web Missing required labels (app.kubernetes.io/name or /version)
LOW mrs-labels-missing-required Deployment/pr-auth Missing required labels (app.kubernetes.io/name or /version)
LOW mrs-labels-missing-required Deployment/pr-long Missing required labels (app.kubernetes.io/name or /version)
LOW mrs-allow-priv-esc-unset Deployment/pr-web allowPrivilegeEscalation not set — defaults can allow privilege escalation
LOW mrs-readonly-rootfs Deployment/pr-web readOnlyRootFilesystem not enabled — writable container root filesystem
LOW mrs-seccomp-unset Deployment/pr-web seccompProfile not set — relies on cluster default
LOW mrs-allow-priv-esc-unset Deployment/pr-auth allowPrivilegeEscalation not set — defaults can allow privilege escalation
LOW mrs-readonly-rootfs Deployment/pr-auth readOnlyRootFilesystem not enabled — writable container root filesystem
LOW mrs-seccomp-unset Deployment/pr-auth seccompProfile not set — relies on cluster default
LOW mrs-allow-priv-esc-unset Deployment/pr-long allowPrivilegeEscalation not set — defaults can allow privilege escalation
LOW mrs-readonly-rootfs Deployment/pr-long readOnlyRootFilesystem not enabled — writable container root filesystem
LOW mrs-seccomp-unset Deployment/pr-long seccompProfile not set — relies on cluster default
MEDIUM mrs-diff-resource-removed ScaledObject/rtzk-pass/rtzk-pass Resource removed from manifest
MEDIUM mrs-diff-resource-removed TriggerAuthentication/rtzk/rtzk-vault Resource removed from manifest
INFO mrs-diff-resource-added ScaledObject/rtzk-pass/pr-second Resource added to manifest
INFO mrs-diff-resource-added Deployment/rtzk-pass/pr-web Resource added to manifest
INFO mrs-diff-resource-added ScaledObject/rtzk-pass/pr-taken Resource added to manifest
INFO mrs-diff-resource-added ScaledObject/rtzk-pass/pr-ghost Resource added to manifest
INFO mrs-diff-resource-added Deployment/rtzk-pass/pr-auth Resource added to manifest
INFO mrs-diff-resource-added ScaledObject/rtzk-pass/pr-auth Resource added to manifest
INFO mrs-diff-resource-added ScaledObject/istio-system/pr-istiod Resource added to manifest
INFO mrs-diff-resource-added ScaledObject/rtz-gate-demo/pr-cap Resource added to manifest
INFO mrs-diff-resource-added ScaledObject/rtzk-pass/pr-name-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa Resource added to manifest
INFO mrs-diff-resource-added TriggerAuthentication/rtzk-pass/pr-file Resource added to manifest
INFO mrs-diff-resource-added Deployment/rtzk-pass/pr-long Resource added to manifest

Capacity fit (3)

  • Fits Deployment rtzk-pass/pr-web at 1 replica(s) (spec.replicas) on the cluster's last sync 30dd57e2-2fe6-43f7-a04c-26ab97015b15: all 1 pod(s) place on nodes there now (spot); judged once the rollout has finished — its surge pods are not placed. Room left after it: 1165m cpu, 2.2Gi memory, 2 pod slots on 3 nodes.
  • Fits Deployment rtzk-pass/pr-auth at 1 replica(s) (spec.replicas) on the cluster's last sync 30dd57e2-2fe6-43f7-a04c-26ab97015b15: all 1 pod(s) place on nodes there now (spot); judged once the rollout has finished — its surge pods are not placed. Room left after it: 1165m cpu, 2.2Gi memory, 2 pod slots on 3 nodes.
  • Fits Deployment rtzk-pass/pr-long at 1 replica(s) (spec.replicas) on the cluster's last sync 30dd57e2-2fe6-43f7-a04c-26ab97015b15: all 1 pod(s) place on nodes there now (spot); judged once the rollout has finished — its surge pods are not placed. Room left after it: 1165m cpu, 2.2Gi memory, 2 pod slots on 3 nodes.

Add-on findings · KEDA (9)

Severity Rule Resource Title
HIGH wrs-addon-keda-target-has-other-scaledobject ScaledObject/rtzk-pass/pr-second Workload is already scaled by another KEDA ScaledObject
HIGH wrs-addon-keda-hpa-name-taken ScaledObject/rtzk-pass/pr-taken KEDA HPA name already belongs to another ScaledObject
HIGH wrs-addon-keda-scale-target-missing ScaledObject/rtzk-pass/pr-ghost KEDA ScaledObject scales a workload that does not exist
HIGH wrs-addon-keda-auth-ref-missing ScaledObject/rtzk-pass/pr-auth KEDA trigger references an authentication that does not exist
HIGH wrs-addon-keda-target-has-other-hpa ScaledObject/istio-system/pr-istiod Workload is already scaled by another HorizontalPodAutoscaler
MEDIUM wrs-addon-keda-max-below-current ScaledObject/rtz-gate-demo/pr-cap KEDA maxReplicaCount is below the replicas the workload runs now
MEDIUM wrs-addon-keda-scaledobject-removed ScaledObject/rtzk-pass/rtzk-pass KEDA ScaledObject removed
MEDIUM wrs-addon-keda-paused ScaledObject/rtzk/rtzk-stan KEDA autoscaling paused
HIGH wrs-addon-keda-deleted-auth-still-referenced TriggerAuthentication/rtzk/rtzk-vault KEDA authentication deleted while scalers still use it
What changed and how to fix it
  • wrs-rs-unset · Deployment/pr-web — Set CPU and memory requests on all containers to enable proper scheduling and QoS.
  • wrs-rs-unset · Deployment/pr-auth — Set CPU and memory requests on all containers to enable proper scheduling and QoS.
  • wrs-rs-unset · Deployment/pr-long — Set CPU and memory requests on all containers to enable proper scheduling and QoS.
  • wrs-spec-no-readiness · Deployment/pr-web — Add a readinessProbe to ensure traffic is only sent to pods that are ready to serve requests.
  • wrs-spec-no-readiness · Deployment/pr-auth — Add a readinessProbe to ensure traffic is only sent to pods that are ready to serve requests.
  • wrs-spec-no-readiness · Deployment/pr-long — Add a readinessProbe to ensure traffic is only sent to pods that are ready to serve requests.
  • wrs-spec-no-liveness · Deployment/pr-web — Add a livenessProbe so Kubernetes can detect and restart stuck or deadlocked pods.
  • wrs-spec-no-liveness · Deployment/pr-auth — Add a livenessProbe so Kubernetes can detect and restart stuck or deadlocked pods.
  • wrs-spec-no-liveness · Deployment/pr-long — Add a livenessProbe so Kubernetes can detect and restart stuck or deadlocked pods.
  • wrs-rs-nolimit · Deployment/pr-web — Set resource limits to prevent this workload from consuming excessive node resources.
  • wrs-rs-nolimit · Deployment/pr-auth — Set resource limits to prevent this workload from consuming excessive node resources.
  • wrs-rs-nolimit · Deployment/pr-long — Set resource limits to prevent this workload from consuming excessive node resources.
  • mrs-runasroot-unspecified · Deployment/pr-web — No securityContext.runAsNonRoot:true is set; the container may run as root depending on the image. Set runAsNonRoot:true (and a non-zero runAsUser) to guarantee non-root.
  • mrs-runasroot-unspecified · Deployment/pr-auth — No securityContext.runAsNonRoot:true is set; the container may run as root depending on the image. Set runAsNonRoot:true (and a non-zero runAsUser) to guarantee non-root.
  • mrs-runasroot-unspecified · Deployment/pr-long — No securityContext.runAsNonRoot:true is set; the container may run as root depending on the image. Set runAsNonRoot:true (and a non-zero runAsUser) to guarantee non-root.
  • mrs-labels-missing-required · Deployment/pr-web — Add app.kubernetes.io/name and app.kubernetes.io/version labels to the workload metadata. These labels are required for workload tracking and are used by observability, cost-allocation, and upgrade-intelligence tools.
  • mrs-labels-missing-required · Deployment/pr-auth — Add app.kubernetes.io/name and app.kubernetes.io/version labels to the workload metadata. These labels are required for workload tracking and are used by observability, cost-allocation, and upgrade-intelligence tools.
  • mrs-labels-missing-required · Deployment/pr-long — Add app.kubernetes.io/name and app.kubernetes.io/version labels to the workload metadata. These labels are required for workload tracking and are used by observability, cost-allocation, and upgrade-intelligence tools.
  • mrs-allow-priv-esc-unset · Deployment/pr-web — Explicitly set securityContext.allowPrivilegeEscalation: false. When unset, the effective value can permit privilege escalation depending on other settings.
  • mrs-readonly-rootfs · Deployment/pr-web — Set securityContext.readOnlyRootFilesystem: true and mount writable paths as explicit emptyDir/volumes. A read-only root limits tampering and persistence after a compromise.
  • mrs-seccomp-unset · Deployment/pr-web — Set seccompProfile.type: RuntimeDefault at pod or container level so syscall filtering is explicit and not dependent on cluster defaults.
  • mrs-allow-priv-esc-unset · Deployment/pr-auth — Explicitly set securityContext.allowPrivilegeEscalation: false. When unset, the effective value can permit privilege escalation depending on other settings.
  • mrs-readonly-rootfs · Deployment/pr-auth — Set securityContext.readOnlyRootFilesystem: true and mount writable paths as explicit emptyDir/volumes. A read-only root limits tampering and persistence after a compromise.
  • mrs-seccomp-unset · Deployment/pr-auth — Set seccompProfile.type: RuntimeDefault at pod or container level so syscall filtering is explicit and not dependent on cluster defaults.
  • mrs-allow-priv-esc-unset · Deployment/pr-long — Explicitly set securityContext.allowPrivilegeEscalation: false. When unset, the effective value can permit privilege escalation depending on other settings.
  • mrs-readonly-rootfs · Deployment/pr-long — Set securityContext.readOnlyRootFilesystem: true and mount writable paths as explicit emptyDir/volumes. A read-only root limits tampering and persistence after a compromise.
  • mrs-seccomp-unset · Deployment/pr-long — Set seccompProfile.type: RuntimeDefault at pod or container level so syscall filtering is explicit and not dependent on cluster defaults.
  • mrs-diff-resource-removed · ScaledObject/rtzk-pass/rtzk-pass — ScaledObject rtzk-pass/rtzk-pass was removed from the manifest. Verify this deletion is intentional.
    Fix: A resource present in the base manifest is absent in the head. Verify this deletion is intentional and that no other workloads depend on it.
  • mrs-diff-resource-removed · TriggerAuthentication/rtzk/rtzk-vault — TriggerAuthentication rtzk/rtzk-vault was removed from the manifest. Verify this deletion is intentional.
    Fix: A resource present in the base manifest is absent in the head. Verify this deletion is intentional and that no other workloads depend on it.
  • mrs-diff-resource-added · ScaledObject/rtzk-pass/pr-second — ScaledObject rtzk-pass/pr-second was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · Deployment/rtzk-pass/pr-web — Deployment rtzk-pass/pr-web was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · ScaledObject/rtzk-pass/pr-taken — ScaledObject rtzk-pass/pr-taken was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · ScaledObject/rtzk-pass/pr-ghost — ScaledObject rtzk-pass/pr-ghost was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · Deployment/rtzk-pass/pr-auth — Deployment rtzk-pass/pr-auth was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · ScaledObject/rtzk-pass/pr-auth — ScaledObject rtzk-pass/pr-auth was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · ScaledObject/istio-system/pr-istiod — ScaledObject istio-system/pr-istiod was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · ScaledObject/rtz-gate-demo/pr-cap — ScaledObject rtz-gate-demo/pr-cap was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · ScaledObject/rtzk-pass/pr-name-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa — ScaledObject rtzk-pass/pr-name-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · TriggerAuthentication/rtzk-pass/pr-file — TriggerAuthentication rtzk-pass/pr-file was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • mrs-diff-resource-added · Deployment/rtzk-pass/pr-long — Deployment rtzk-pass/pr-long was added to the manifest.
    Fix: A new resource is present in the head manifest that was not in the base. Informational: verify the addition is intentional.
  • wrs-addon-keda-target-has-other-scaledobject · ScaledObject/rtzk-pass/pr-second — ScaledObject rtzk-pass/pr-second: Deployment rtzk-pass is already scaled by ScaledObject rtzk-pass (in the cluster).
    Fix: Merge the triggers into the existing ScaledObject, or delete it in the same change. A workload can have one ScaledObject: KEDA's admission webhook refuses the second, and without the webhook their HPAs fight over the replica count.
  • wrs-addon-keda-hpa-name-taken · ScaledObject/rtzk-pass/pr-taken — ScaledObject rtzk-pass/pr-taken: HPA keda-hpa-rtzk-pass already belongs to ScaledObject rtzk-pass.
    Fix: Choose a different spec.advanced.horizontalPodAutoscalerConfig.name (or a different ScaledObject name when none is set): another ScaledObject in the namespace already manages an HPA of that name. KEDA's admission webhook refuses the apply; with the webhook switched off the two ScaledObjects overwrite one HPA.
  • wrs-addon-keda-scale-target-missing · ScaledObject/rtzk-pass/pr-ghost — ScaledObject rtzk-pass/pr-ghost: Deployment rtzk-pass/does-not-exist is neither in this change nor in the cluster.
    Fix: Fix spec.scaleTargetRef (kind, name, apiVersion) or ship the workload in the same change. With a cpu or memory trigger KEDA's admission webhook reads the workload and refuses the apply; otherwise the ScaledObject applies, goes Ready=False (ScaledObjectCheckFailed) and gets no HPA.
  • wrs-addon-keda-auth-ref-missing · ScaledObject/rtzk-pass/pr-auth — ScaledObject rtzk-pass/pr-auth: the trigger authentication TriggerAuthentication no-such-auth is neither in this change nor in the cluster.
    Fix: Create the TriggerAuthentication (same namespace) or ClusterTriggerAuthentication the trigger names, or fix authenticationRef.name / kind. Until it exists KEDA logs "error getting triggerAuth" and builds the scaler without its authentication parameters: a scaler that needs credentials or a host from it fails to connect and the trigger errors.
  • wrs-addon-keda-target-has-other-hpa · ScaledObject/istio-system/pr-istiod — ScaledObject istio-system/pr-istiod: Deployment istiod-upstream-1-29-3 is already scaled by HorizontalPodAutoscaler istiod-upstream-1-29-3 (in the cluster).
    Fix: Delete the existing HPA first, or take it over: annotate the ScaledObject scaledobject.keda.sh/transfer-hpa-ownership: "true" AND set spec.advanced.horizontalPodAutoscalerConfig.name to that HPA's name. KEDA's admission webhook refuses the ScaledObject otherwise; without the webhook two HPAs fight over the replica count.
  • wrs-addon-keda-max-below-current · ScaledObject/rtz-gate-demo/pr-cap — ScaledObject rtz-gate-demo/pr-cap: Deployment gate-web runs 2 replicas now; maxReplicaCount 1 scales it in by 1 as soon as this lands.
    Fix: Check the new maxReplicaCount: as soon as the change lands KEDA's HPA scales the workload in to that number, removing pods that serve traffic now. Raise it, or lower it in steps.
  • wrs-addon-keda-scaledobject-removed · ScaledObject/rtzk-pass/rtzk-pass — ScaledObject rtzk-pass/rtzk-pass: Deployment rtzk-pass stops autoscaling: KEDA deletes its HPA and the workload stays at the replica count it has at that moment.
    Fix: Confirm the workload should stop autoscaling: without its ScaledObject KEDA deletes the HPA and the workload stays at its current replica count (or the pre-KEDA count when advanced.restoreToOriginalReplicaCount is true) until someone scales it.
  • wrs-addon-keda-paused · ScaledObject/rtzk/rtzk-stan — ScaledObject rtzk/rtzk-stan: autoscaling.keda.sh/paused: "true" freezes the replica count.
    Fix: Confirm the pause is intended and temporary: autoscaling.keda.sh/paused freezes the replica count where it is (paused-replicas pins it to a number, paused-scale-in / paused-scale-out freeze one direction). Remove the annotation to resume.
  • wrs-addon-keda-deleted-auth-still-referenced · TriggerAuthentication/rtzk/rtzk-vault — TriggerAuthentication rtzk/rtzk-vault: deleted, but 1 object(s) still use it: ScaledObject rtzk/rtzk-vault (in the cluster).
    Fix: Keep the TriggerAuthentication / ClusterTriggerAuthentication until no ScaledObject or ScaledJob references it, or repoint those triggers in the same change. Without it KEDA builds every scaler that references it without its authentication parameters, so a scaler that needs credentials or a host from it fails.

Cluster-aware checks: on — 68 runtimez lookup(s) answered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant