Describe the bug
Kubernetes pod cgroups are in the system path /sys/fs/cgroup/kubepods.slice/. The java security policy used by Opensearch does not allow access to read this path. This causes the following error when opensearch attempts to determine certain settings based on available memory:
Caused by: java.lang.SecurityException: Denied OPEN (read) access to file: /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<some uuid>.slice/cri-containerd-<some id>.scope/memory.max, domain: ProtectionDomain (file:/usr/share/opensearch/lib/opensearch-3.7.0.jar <no signer certificates>)
In some older 3.x versions, it was possible to set problematic settings to static values (e.g. knn.memory.circuit_breaker.limit), bypassing the need to read cgroup info. This is no longer appears to be possible in 3.7 with at least one specific setting indices.memory.native_index_buffer_size. Th inability to read the cgroup causes the following error with regards to this setting:
OpenSearchParseException[cannot compute native memory percentage for [indices.memory.native_index_buffer_size]: total physical memory [0] minus JVM heap [1073741824] is non-positive; use an absolute value instead of a percentage]
Rather than continuing to play whackahole with settings, the Opensearch's security policy should be updated to allow read access to the common k8s pod cgroup path.
Related component
Libraries
To Reproduce
- Launch an opensearch 3.5+ container on a k8s cluster
- observe it failing to access the pod cgroup paths
- Container crashes/fails to start
Expected behavior
It is able to access the pod cgroup paths, and starts up successfully
Additional Details
Host/Environment (please complete the following information):
- Platform: AWS EKS
- Openseach: version 3.5 and 3.7
Describe the bug
Kubernetes pod cgroups are in the system path
/sys/fs/cgroup/kubepods.slice/. The java security policy used by Opensearch does not allow access to read this path. This causes the following error when opensearch attempts to determine certain settings based on available memory:Caused by: java.lang.SecurityException: Denied OPEN (read) access to file: /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<some uuid>.slice/cri-containerd-<some id>.scope/memory.max, domain: ProtectionDomain (file:/usr/share/opensearch/lib/opensearch-3.7.0.jar <no signer certificates>)In some older 3.x versions, it was possible to set problematic settings to static values (e.g.
knn.memory.circuit_breaker.limit), bypassing the need to read cgroup info. This is no longer appears to be possible in 3.7 with at least one specific settingindices.memory.native_index_buffer_size. Th inability to read the cgroup causes the following error with regards to this setting:OpenSearchParseException[cannot compute native memory percentage for [indices.memory.native_index_buffer_size]: total physical memory [0] minus JVM heap [1073741824] is non-positive; use an absolute value instead of a percentage]Rather than continuing to play whackahole with settings, the Opensearch's security policy should be updated to allow read access to the common k8s pod cgroup path.
Related component
Libraries
To Reproduce
Expected behavior
It is able to access the pod cgroup paths, and starts up successfully
Additional Details
Host/Environment (please complete the following information):