Add configuration guide & generate script updates

CONFIGURATION.md

@@ -0,0 +1,173 @@
+# Batch Queue Configuration
+
+## Configuration Parameters
+
+| Parameter | Purpose | Valid Range | Typical Values |
+|-----------|---------|-------------|----------------|
+| `calc_interval` | How often to recalculate all priorities | > 0 | `5m` - `30m` |
+| `max_age` | Maximum age for highest adjustment | > 0 | `24h` - `72h` |
+| `half_life` | Exponential decay rate for usage | > 0 | `1h` - `24h` |
+| `max_size` | Maximum job size for highest adjustment| > 0 | `1000` - `10000` |
+| `age_weight` | Priority boost for waiting jobs | any | `10` - `50` |
+| `size_weight` | Priority boost for job size preference | any | `10` - `50` |
+| `usage_weight` | Priority boost for resource fairness | any | `20` - `60` |
+
+### Understanding the Priority Formula
+
+The Dynamic Priority Queue calculates priority as:
+
+```
+priority = base_priority 
+         + (age_adjustment × age_weight)
+         + (usage_adjustment × usage_weight) 
+         + (size_adjustment × size_weight)
+```
+
+### Usage, Decay, and Garbage Collection
+tbd
+
+## Common Configurations
+
+### First-In-First-Out (FIFO) 
+
+You want jobs to run in submission order with minimal overhead.
+- No fairness requirements
+- Jobs run in submission order
+- Minimal computational overhead
+
+**Configuration**
+```hcl
+server {
+  batch_queue {
+    type        = "fifo"
+  }
+}
+```
+
+**Warnings**:
+- No protection against monopolization, because there are no tenants or priority
+
+---
+
+### Resource Fairness 
+
+Enforce strict fair sharing based on resource consumption.
+- Tenants who have used more resources get lower priority
+- Usage decays exponentially with according to half-life
+- Jobs from light users almost always run first
+- Age provides escape valve (example: max 48h wait)
+- With default `batch_eval_gc_threshold=24h`: Usage fully removed after 24h (6.25% of original remains at GC time)
+
+Use when:
+- You have multiple tenants with equal rights, where you need to prevent resource monopolization
+- Resource consumption varies significantly between tenants
+- Fairness is more important than urgency
+
+**Configuration**
+```hcl
+server {
+  batch_queue {
+    type        = "dynamic_priority"
+    ...
+
+    config = {
+      calc_interval = "10m"
+      half_life     = "6h"
+      max_age       = "48h"
+      max_size      = 5000
+      age_weight    = 10
+      usage_weight  = 80
+      size_weight   = 0
+    }
+  }
+}
+```
+
+**Warnings**
+- Cold start: All tenants at zero resource usage means that this queue behaves like FIFO initially
+- Urgent jobs from heavy users may wait too long
+- New tenants flood the queue before building history
+- When a tenant submits one large job vs many tiny jobs there are no penalty differences
+- When a heavy user submits critical high-priority job, it'll wait behind light users
+- If `half_life` is too long (>12h), penalties persist and then disappear suddenly at GC threshold
+- If `usage_weight` is too high and `age_weight` is too low, jobs can wait indefinitely
+
+---
+
+### Priority Lanes
+
+Respect job priority over fairness.
+- Base priority (ex. 0-100) dominates
+- Light fairness touch (usage_weight=20)
+- High-priority jobs run quickly even from heavy users
+- Usage penalties forgive slowly (half_life=12h)
+
+Use when:
+- Urgency matters more than fairness
+
+**Configuration**
+```hcl
+server {
+  batch_queue {
+    type        = "dynamic_priority"
+    ...
+
+    config = {
+      calc_interval = "10m"
+      half_life     = "12h"
+      max_age       = "72h"
+      max_size      = 5000
+      age_weight    = 20
+      usage_weight  = 20
+      size_weight   = 0
+    }
+  }
+}
+```
+
+**Warnings**
+- The light fairness enforcement means that heavy users can monopolize the queue
+    - If `usage_weight` too low, monopolization is possible
+- Requires policy enforcement on job priorities outside of the queue (Gaming is possible if users can set their own priorities)
+
+---
+
+### Backfill Small Jobs
+
+Maximize cluster utilization by preferring small jobs that can fit in gaps.
+- Small jobs get priority boost, large jobs get deprioritized
+- Still enforces fairness (usage_weight=30)
+- Age_weight=20 prevents indefinite starvation
+
+Use when: 
+- Mix of large and small jobs
+- Want to maximize throughput based on cluster resources
+
+**Configuration**
+```hcl
+server {
+  batch_queue {
+    type        = "dynamic_priority"
+    ...
+
+    config = {
+      calc_interval = "5m"
+      half_life     = "3h"
+      max_age       = "24h"
+      max_size      = 5000
+      age_weight    = 20
+      usage_weight  = 30
+      size_weight   = 30
+    }
+  }
+}
+```
+
+**Warnings**
+- Large jobs may wait a very long time, while small jobs can flood the queue
+    - `age_weight` is set, but it might not be able to overcome size penalty fast enough
+- Doesn't guarantee backfill placement (scheduler still makes placement decisions)
+- If all jobs are large, `size_weight` has no effect
+- If `max_size` is misconfigured (too small), all jobs look "large"
+
+---

examples/dynamic_priority/README.md

@@ -6,12 +6,12 @@ The provided server config can be run in dev mode via:
 nomad agent -dev -config /path/to/examples/dynamic_priority/server_config.hcl
 ```
 
-Use `generate.sh <id> <num_jobs> <cpu> <mem>` to generate some test jobs:
+Use `generate.sh [options] <id> <num_jobs> <cpu> <mem>` to generate some test jobs:
 
  - `id`: Job ID
  - `num_jobs`: How many jobspecs to generate
- - `cpu`: MHz each job should require
- - `mem`: MB of memory each job should require
+ - `cpu`: Max MHz each job should require
+ - `mem`: Max MB of memory each job should require
 
 Ex. `./generate foo 10 10000 50000` and `./generate bar 10 10000 50000`
 

examples/dynamic_priority/generate.sh

@@ -4,37 +4,71 @@
 
 set -euo pipefail
 
-if [[ $# -ne 4 ]]; then
-    echo "Usage: $0 <id> <num_jobs> <cpu> <mem>"
+OUTPUT_DIR="$PWD"
+NUM_TENANTS=1
+STRICT=false
+
+usage() {
+    echo "Usage: $0 [options] <tenant_id> <num_jobs> <cpu> <mem>"
     echo "Example: $0 test 10 1000 4000"
+    echo "Options:"
+    echo "  -o <output_dir>   Output directory (default: current directory)"
+    echo "  -p                Include random priority in jobs"
+    echo "  -s                Use strict resource allocation for all jobs"
+    echo "  -t <num_tenants>  Number of tenants with tenant_id prefix (default: 1)"
+    echo "  -h                Show this help message"
     exit 1
+}
+
+while getopts "o:t:psh" opt; do
+    case $opt in
+        o) OUTPUT_DIR="$OPTARG" ;;
+        p) WITH_PRIORITY=true ;;
+        s) STRICT=true ;;
+        t) NUM_TENANTS="$OPTARG" ;;
+        h) usage ;;
+        *) usage ;;
+    esac
+done
+shift $((OPTIND-1))
+
+
+if [[ $# -ne 4 ]]; then
+  usage
 fi
 
 id="$1"
 num_jobs="$2"
 cpu="$3"
 mem="$4"
 
-mkdir -p $PWD/jobs
+mkdir -p "$OUTPUT_DIR/jobs"
 
-for ((index=0; index<num_jobs; index++)); do
-  filename="${id}_${index}.nomad.hcl"
+for ((j=0; j<NUM_TENANTS; j++)); do
+  if [[ $j -eq 0 ]]; then
+    tenant="${id}"
+  else
+    tenant="${id}_${j}"
+  fi
+  for ((index=0; index<num_jobs; index++)); do
+    filename="${tenant}_${index}.nomad.hcl"
 
-  cat > $PWD/jobs/$filename <<EOF
-job "${id}_${index}" {
+  cat > "$OUTPUT_DIR/jobs/$filename" <<EOF
+job "${tenant}_${index}" {
   type = "batch"
 
   meta {
-    id = "${id}"
+    id = "${tenant}"
   }
 
   group "group" {
+    $([[ -n "${WITH_PRIORITY:-}" ]] && echo priority = $((1 + RANDOM % 100)))
     task "sleep" {
       driver = "docker"
 
       resources {
-        cpu    = ${cpu}
-        memory = ${mem}
+        cpu    = $([[ -n "${STRICT:-}" ]] && echo $((10 + RANDOM % (cpu - 10 + 1))) || echo $((cpu)))
+        memory = $([[ -n "${STRICT:-}" ]] && echo $((10 + RANDOM % (mem - 10 + 1))) || echo $((mem)))
       }
 
       config {
@@ -47,4 +81,5 @@ job "${id}_${index}" {
 }
 EOF
 
+  done
 done