Nomad can place many workloads very quickly due to it's shared state and parallelized schedulers. This means that clusters that run many batch jobs can place those workloads quickly. However, clusters that often run more batch workloads than they have capacity for end up with many batch jobs in the "pending" state, waiting for cluster capacity to free up so they can be placed.
When these batch jobs take a while to complete, Nomad users often want to have some level of control of which batch job is placed next. Currently, Nomad offers no real guarantees here. Job priorities exist, but they only control the order in which Nomad evaluations are popped off Nomad's internal queueing system or for preempting (stopping) other running jobs. The priority parameter does have some impact on which job will be scheduled next, but because of Nomad's parallelized schedulers, it can provide no guarantees.
To solve this, operators often build an external queueing system that sits infront of Nomad. Users then submit jobs to this queue, which runs custom logic for both monitoring Nomad and selecting the next job to submit to Nomad.
Nomad Batch Queues are a builtin way to configure native queue implementations for controlling the placement of batch workloads. They give Nomad operators direct control over the order in which batch jobs are placed in their cluster.
When looking at the flow of evaluation (refer to how-scheduling-works), the batch queue sits right before the evaluation broker but after jobs and their evaluations are persisted via Raft to a Nomad server's in memory state. A batch queue can efficiently monitor Nomad and submit batch job evaluations to the evaluation broker when it is time to be placed in the cluster. When this happens, a scheduler will pick up the evaluation and attempt to place it. The batch queue will wait until placement is successful until attempting to place another job. By doing this, a queue can guarantee the order that jobs are placed.
The batch queue is not a separate scheduler implementation. That means it cannot make placement decisions in Nomad, or evaluate job contraints to see if one job could be placed in the cluster before another. It simply holds units of work (Nomad Batch Job's) and passes them onto Nomad via the rules defined in it's implementation (refer to Queue Types). Once a Job has been given to a Nomad scheduler, it is no longer the batch queues responsibility. If another job in the queue needs to be placed, it must wait until Nomad has finished placing the previous job.
Given this, there's 2 types of batch jobs that may not be a good fit for batch queues. Those are:
-
Jobs with many allocations relative to cluster size Depending on the size of the cluster, these jobs can take a long time to finish placement. Often they require previously placed allocations to finish before other allocations in the job can be placed. In these instances, a Nomad user might want to run a faster/smaller job, but will have to wait until the previous job is done being placed. If the larger jobs was broken up into more jobs of fewer allocations, the batch queue release a higher priority job to Nomad without having to wait as long.
-
Jobs with varying contraints These types of jobs should not be placed on the same queue. As mentioned previously, the batch queue does not inspect job constraints. A simple example of this would be we had 2 jobs, both with different OS contraints. There would be no reason to wait for the first job to finish placement because they are being placed on different nodes.
The public RFC explains the initial queue implementation, the Dynamic Priority Qeueue. This queue maintains an internal priority for each job, which is the job's priority +/- other factors such as how long the job has been in the queue, the size of the job, and a "usage" factor. These factors are explained more in the example setup for a Dynamic Priority Queue in the README.md.
The other implementation is a simpler FIFO queue. This queue simply maintains FIFO ordering based on a Job's raft index. It will always ensure that a job with an earlier raft index is placed in the cluster before a job with a greater index.