fix: relax perf_event_paranoid/kptr_restrict for EKS Auto Mode (Bottlerocket)#454
Conversation
On EKS Auto Mode (Bottlerocket), the host kernel ships with perf_event_paranoid=2 and a non-zero kptr_restrict, which causes 'aperf record' to fail with a "Run sudo sysctl -w ..." error from inside the privileged pod. Since the pod is already privileged with hostPID, write the relaxed values directly to the host's /proc/sys before invoking aperf record.
Per review feedback on PR aws#454: capture the host's original /proc/sys/kernel/perf_event_paranoid and /proc/sys/kernel/kptr_restrict values before relaxing them, and restore them on EXIT/INT/TERM via a trap so the host is left as we found it even if aperf record fails or the pod is signaled.
|
Updated in b047bbd to address @salvatoredipietro's feedback: capture the original |
Live test on EKS Auto Mode (Bottlerocket) showed that the EXIT trap alone is unreliable: the script blocks on 'sleep 7200' after record, and the SIGKILL that follows 'kubectl delete pod' (after grace period) prevents the trap from running, leaving the host with relaxed perf_event_paranoid/kptr_restrict. Restore the original values explicitly as soon as 'aperf record' returns, then clear the trap. The trap remains as a fallback for the case where 'aperf record' itself crashes before we reach this point. Verified end-to-end on a Bottlerocket EKS Auto node: host sysctls go 2->-1 during record and back to 2 after the script restores them, even after the pod is deleted.
|
Tested live on a Bottlerocket EKS Auto Mode 1.31 node. First run of the trap-only approach surfaced a real issue: the script blocks on Pushed 8261189 which restores the originals explicitly as soon as Re-verified end-to-end: orig captured as |
Great, Thanks! |
2 participants
Issue #, if available: N/A
Description of changes:
Tested on a fresh EKS Auto Mode 1.31 cluster (Bottlerocket workload node). PSS / privileged / hostPID / hostPath all work on default namespaces. One gap on Auto specifically: Bottlerocket ships with
perf_event_paranoid=2and a non-zerokptr_restrict, and the plugin doesn't relax them inside the pod — soaperf recordfails with a "Run sudo sysctl -w …" error.This PR has the plugin write
-1to/proc/sys/kernel/perf_event_paranoidand0to/proc/sys/kernel/kptr_restrictbefore invokingaperf record. The pod is already privileged withhostPID, so it can write the host's/proc/sys. The writes are best-effort (|| echo "Warning: …") so distros that don't need them, or restrict them further, don't break the recording flow.