diff --git a/CONFIGURATION.md b/CONFIGURATION.md new file mode 100644 index 0000000..5e8a56b --- /dev/null +++ b/CONFIGURATION.md @@ -0,0 +1,173 @@ +# Batch Queue Configuration + +## Configuration Parameters + +| Parameter | Purpose | Valid Range | Typical Values | +|-----------|---------|-------------|----------------| +| `calc_interval` | How often to recalculate all priorities | > 0 | `5m` - `30m` | +| `max_age` | Maximum age for highest adjustment | > 0 | `24h` - `72h` | +| `half_life` | Exponential decay rate for usage | > 0 | `1h` - `24h` | +| `max_size` | Maximum job size for highest adjustment| > 0 | `1000` - `10000` | +| `age_weight` | Priority boost for waiting jobs | any | `10` - `50` | +| `size_weight` | Priority boost for job size preference | any | `10` - `50` | +| `usage_weight` | Priority boost for resource fairness | any | `20` - `60` | + +### Understanding the Priority Formula + +The Dynamic Priority Queue calculates priority as: + +``` +priority = base_priority + + (age_adjustment × age_weight) + + (usage_adjustment × usage_weight) + + (size_adjustment × size_weight) +``` + +### Usage, Decay, and Garbage Collection +tbd + +## Common Configurations + +### First-In-First-Out (FIFO) + +You want jobs to run in submission order with minimal overhead. +- No fairness requirements +- Jobs run in submission order +- Minimal computational overhead + +**Configuration** +```hcl +server { + batch_queue { + type = "fifo" + } +} +``` + +**Warnings**: +- No protection against monopolization, because there are no tenants or priority + +--- + +### Resource Fairness + +Enforce strict fair sharing based on resource consumption. +- Tenants who have used more resources get lower priority +- Usage decays exponentially with according to half-life +- Jobs from light users almost always run first +- Age provides escape valve (example: max 48h wait) +- With default `batch_eval_gc_threshold=24h`: Usage fully removed after 24h (6.25% of original remains at GC time) + +Use when: +- You have multiple tenants with equal rights, where you need to prevent resource monopolization +- Resource consumption varies significantly between tenants +- Fairness is more important than urgency + +**Configuration** +```hcl +server { + batch_queue { + type = "dynamic_priority" + ... + + config = { + calc_interval = "10m" + half_life = "6h" + max_age = "48h" + max_size = 5000 + age_weight = 10 + usage_weight = 80 + size_weight = 0 + } + } +} +``` + +**Warnings** +- Cold start: All tenants at zero resource usage means that this queue behaves like FIFO initially +- Urgent jobs from heavy users may wait too long +- New tenants flood the queue before building history +- When a tenant submits one large job vs many tiny jobs there are no penalty differences +- When a heavy user submits critical high-priority job, it'll wait behind light users +- If `half_life` is too long (>12h), penalties persist and then disappear suddenly at GC threshold +- If `usage_weight` is too high and `age_weight` is too low, jobs can wait indefinitely + +--- + +### Priority Lanes + +Respect job priority over fairness. +- Base priority (ex. 0-100) dominates +- Light fairness touch (usage_weight=20) +- High-priority jobs run quickly even from heavy users +- Usage penalties forgive slowly (half_life=12h) + +Use when: +- Urgency matters more than fairness + +**Configuration** +```hcl +server { + batch_queue { + type = "dynamic_priority" + ... + + config = { + calc_interval = "10m" + half_life = "12h" + max_age = "72h" + max_size = 5000 + age_weight = 20 + usage_weight = 20 + size_weight = 0 + } + } +} +``` + +**Warnings** +- The light fairness enforcement means that heavy users can monopolize the queue + - If `usage_weight` too low, monopolization is possible +- Requires policy enforcement on job priorities outside of the queue (Gaming is possible if users can set their own priorities) + +--- + +### Backfill Small Jobs + +Maximize cluster utilization by preferring small jobs that can fit in gaps. +- Small jobs get priority boost, large jobs get deprioritized +- Still enforces fairness (usage_weight=30) +- Age_weight=20 prevents indefinite starvation + +Use when: +- Mix of large and small jobs +- Want to maximize throughput based on cluster resources + +**Configuration** +```hcl +server { + batch_queue { + type = "dynamic_priority" + ... + + config = { + calc_interval = "5m" + half_life = "3h" + max_age = "24h" + max_size = 5000 + age_weight = 20 + usage_weight = 30 + size_weight = 30 + } + } +} +``` + +**Warnings** +- Large jobs may wait a very long time, while small jobs can flood the queue + - `age_weight` is set, but it might not be able to overcome size penalty fast enough +- Doesn't guarantee backfill placement (scheduler still makes placement decisions) +- If all jobs are large, `size_weight` has no effect +- If `max_size` is misconfigured (too small), all jobs look "large" + +--- \ No newline at end of file diff --git a/examples/dynamic_priority/README.md b/examples/dynamic_priority/README.md index 1932e7d..ac79ccd 100644 --- a/examples/dynamic_priority/README.md +++ b/examples/dynamic_priority/README.md @@ -6,12 +6,12 @@ The provided server config can be run in dev mode via: nomad agent -dev -config /path/to/examples/dynamic_priority/server_config.hcl ``` -Use `generate.sh ` to generate some test jobs: +Use `generate.sh [options] ` to generate some test jobs: - `id`: Job ID - `num_jobs`: How many jobspecs to generate - - `cpu`: MHz each job should require - - `mem`: MB of memory each job should require + - `cpu`: Max MHz each job should require + - `mem`: Max MB of memory each job should require Ex. `./generate foo 10 10000 50000` and `./generate bar 10 10000 50000` diff --git a/examples/dynamic_priority/generate.sh b/examples/dynamic_priority/generate.sh index 3043dd4..6e3befb 100755 --- a/examples/dynamic_priority/generate.sh +++ b/examples/dynamic_priority/generate.sh @@ -4,10 +4,37 @@ set -euo pipefail -if [[ $# -ne 4 ]]; then - echo "Usage: $0 " +OUTPUT_DIR="$PWD" +NUM_TENANTS=1 +STRICT=false + +usage() { + echo "Usage: $0 [options] " echo "Example: $0 test 10 1000 4000" + echo "Options:" + echo " -o Output directory (default: current directory)" + echo " -p Include random priority in jobs" + echo " -s Use strict resource allocation for all jobs" + echo " -t Number of tenants with tenant_id prefix (default: 1)" + echo " -h Show this help message" exit 1 +} + +while getopts "o:t:psh" opt; do + case $opt in + o) OUTPUT_DIR="$OPTARG" ;; + p) WITH_PRIORITY=true ;; + s) STRICT=true ;; + t) NUM_TENANTS="$OPTARG" ;; + h) usage ;; + *) usage ;; + esac +done +shift $((OPTIND-1)) + + +if [[ $# -ne 4 ]]; then + usage fi id="$1" @@ -15,26 +42,33 @@ num_jobs="$2" cpu="$3" mem="$4" -mkdir -p $PWD/jobs +mkdir -p "$OUTPUT_DIR/jobs" -for ((index=0; index $PWD/jobs/$filename < "$OUTPUT_DIR/jobs/$filename" <