Skip to content

Fold all conditions into the pending workload query#18

Draft
julik wants to merge 3 commits into
mainfrom
improve-pending-workload-query
Draft

Fold all conditions into the pending workload query#18
julik wants to merge 3 commits into
mainfrom
improve-pending-workload-query

Conversation

@julik
Copy link
Copy Markdown
Contributor

@julik julik commented Sep 26, 2024

When we autoscale based on queue depth, and use scheduler keys and fuses, we want to take into account all the conditions. The idea has two benefits:

  • We should have only 1 scope for selecting the next workload, without having it double. It should be sufficiently efficient to be usable in both contexts (autoscaler metric and dequeue)
  • When we autoscale, the fuses should have effect on that autoscaling. If we fuse out 80% of the workloads, there is no point running 80% of compute allocated to those workloads in the anticipation that they are going to become needed once we de-fuse the workloads back in.

Same for scheduler keys - it is possible to know which scheduler keys are currently known to the system, and there is no reason removed scheduled workloads should still influence the autoscaler.

This is WIP, as we need a decent benchmark which allows us to tweak this scope/query with a realistic queue size matching our real-life setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant