Repository navigation
Conversation
Every runner pod copies its tool cache and sysroot payloads before its job starts, and on a node with a slow disk those copies keep the disk busy while the jobs beside them wait on it. A profile's capability_volume_medium: Memory puts both volumes on a tmpfs so the copies write nothing to the disk; the README says how to raise the pods' memory requests and limits by what the copies hold. The node cache stays a hostPath on the disk.
The init containers that copy the payloads set no limit, so their pages land in the pod's cgroup and count towards eviction against its requests but against no limit; what a job adds to the tool cache counts against the runner container's limit.
Member
Author
|
tmpfs was measured and rejected for the RAM cost: the copies would sit in memory on every node, raising each runner pod's memory request and lowering the fleet's ceiling. Replaced by #132, which copies each payload once per node into a store keyed by image digest. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every runner pod copies its tool-cache and system-library payloads into
emptyDirvolumes before its job starts, so each pod start writes the whole tool set and sysroot to the node's disk. On nodes with a slow consumer SSD those copies alone kept the disk saturated while the jobs beside them waited, and pods spent their first minutes in their init containers.This adds a per-profile
capability_volume_medium. Unset, nothing changes.Memoryputs thetool-cacheandsysrootvolumes on a tmpfs, so the copies write nothing to the disk. The node cache is ahostPathand keeps its place. RAM-backed volumes count against the pod's memory, so the README's new "Where the copies live" section says how to size the profile's requests and limits from a running pod and that the higher request lowers what measured sizing derives. The filter takes the medium as a parameter and the profile validation refuses anything other than the two media the copies can use.