chore(db): pause branching benchmark campaign and capture resume state

[elitan]

summary

Paused linux postgres branching benchmark campaign.
When we resume, rerun everything from start. Do not resume from old artifacts.

Goal:

• single-host linux
• postgres in docker
• branching ops fast (<5s p95)
• runtime performance high
• low storage overhead

cleanup done

Benchmark VM cleanup confirmed.
Current Hetzner servers:

• frost-live
• frost-demo-1771268678

No frost-pg-bench-* server is running.

what is implemented

Research harness + adapters + tracking docs are in benchmarks/:

• run-branching-research-all.sh
• run-branching-ops-bench.sh
• run-branching-runtime-bench.sh
• run-branching-scale-bench.sh
• run-branching-soak-bench.sh
• run-ext4-control-bench.sh
• run-branching-hetzner-batch.sh
• provision-linux-vm.sh
• backends/*.sh
• tracking docs/csv in benchmarks/branching-research-*.md|csv

important scripts (copy/paste)

benchmarks/run-branching-research-all.sh
benchmarks/run-branching-hetzner-batch.sh
benchmarks/provision-linux-vm.sh
benchmarks/run-branching-ops-bench.sh
benchmarks/run-ext4-control-bench.sh
benchmarks/run-branching-runtime-bench.sh
benchmarks/run-branching-scale-bench.sh
benchmarks/run-branching-soak-bench.sh
benchmarks/collect-metrics.sh
benchmarks/parse-results.sh
benchmarks/backends/lvm-thin-adapter-core.sh
benchmarks/backends/lvm-thin-ext4-adapter.sh
benchmarks/backends/lvm-thin-meta-adapter.sh
benchmarks/backends/zfs-clone-adapter.sh
benchmarks/backends/btrfs-subvolume-adapter.sh
benchmarks/backends/xfs-reflink-adapter.sh
benchmarks/backends/backend-contract.sh

# full campaign (from scratch)
./benchmarks/run-branching-research-all.sh

# phase-1 ops only
./benchmarks/run-branching-ops-bench.sh

# ext4 baseline only
./benchmarks/run-ext4-control-bench.sh

# runtime only
./benchmarks/run-branching-runtime-bench.sh

# scale only
./benchmarks/run-branching-scale-bench.sh

# soak only
./benchmarks/run-branching-soak-bench.sh

key fixes already made

• fixed CSV parse crash in ops bench (run-branching-ops-bench.sh) by sanitizing multiline/comma error text
• fixed lvm prepare issues (backends/lvm-thin-adapter-core.sh):
  • enforce min LV size for pgbench init
  • save state earlier so cleanup is reliable
• fixed runtime runner to skip failed backends instead of crashing entire batch (run-branching-runtime-bench.sh)
• added phase-1 result override in orchestrator (exists, but do not use for next campaign)

previous run outputs (reference only)

These are historical signals only. Do not use them in final decision package for the next campaign.

Reference dirs:

• benchmarks/results/20260227-085716-branching-research-ops/
• benchmarks/results/20260227-093442-ext4-control/
• benchmarks/results/20260227-120250-branching-research-all/

current blocker to fix before rerun

run-branching-research-all.sh can hang in phase-1 wrapper after remote completion.

Symptom:

• remote ops-gates.csv exists
• local process still stuck on SSH command in run-branching-hetzner-batch.sh

Likely area:

• SSH lifecycle / remote command return handling in run-branching-hetzner-batch.sh

restart plan (from scratch)

1. Fix SSH hang behavior in run-branching-hetzner-batch.sh:
   • timeout around remote execution
   • completion marker check
   • forced teardown path
2. Start a new full run from phase-1, no overrides:
   • ./benchmarks/run-branching-research-all.sh
3. Let full campaign complete in order:
   • phase1 ops
   • phase2 ext4 control
   • phase2 runtime
   • phase2 scale
   • phase3 ext4 control
   • phase3 runtime
   • phase3 scale
   • phase4 soak (24h)
   • decision package
4. Use only the fresh run artifacts for final ranking/report.
5. Verify benchmark VM auto-delete after each batch and at end.

done when

• full campaign completes from phase-1 without artifact reuse
• final report generated in new pipeline dir
• no leftover benchmark VM in Hetzner

[elitan]

adding requested code snapshots here so we can rebuild after reset.

run-branching-ops-bench.sh (csv sanitize fix)

function csv_sanitize() {
  printf '%s' "$1" | tr '\r\n,' ';;;'
}

function append_raw() {
  local backend="$1"
  local parent_size="$2"
  local operation="$3"
  local repeat="$4"
  local latency_ms="$5"
  local success="$6"
  local error="$7"
  printf "%s,%s,%s,%s,%s,%s,%s,%s\n" "$(now_iso)" "$(csv_sanitize "$backend")" "$(csv_sanitize "$parent_size")" "$(csv_sanitize "$operation")" "$(csv_sanitize "$repeat")" "$(csv_sanitize "$latency_ms")" "$(csv_sanitize "$success")" "$(csv_sanitize "$error")" >> "$raw_csv"
}

backends/lvm-thin-adapter-core.sh (lvm parent init fix)

function cmd_prepare_parent() {
  local size_tag="${1:-}"
  [ -z "$size_tag" ] && { echo "size-tag required"; exit 1; }
  cleanup_all || true
  init_state_dir
  local size_bytes
  size_bytes="$(backend_size_to_bytes "$size_tag")"
  local size_gib
  size_gib="$(backend_size_to_human_gib "$size_bytes")"
  local min_main_gib
  min_main_gib="${LVM_MAIN_MIN_GIB:-16}"
  if [ "$size_gib" -lt "$min_main_gib" ]; then
    size_gib="$min_main_gib"
  fi

  VG_NAME="${LVM_VG_NAME:-branchingvg}"
  POOL_NAME="${LVM_POOL_NAME:-branchingpool}"
  META_NAME="${LVM_META_NAME:-branchingmeta}"
  MAIN_SIZE="${size_gib}G"
  BRANCH_ROOT="$STATE_DIR/branches"
  POOL_IMG="$STATE_DIR/pool.img"

  mkdir -p "$BRANCH_ROOT"
  local pool_total_gib="${LVM_POOL_TOTAL_GIB:-220}"
  truncate -s "${pool_total_gib}G" "$POOL_IMG"
  LOOP_DEV="$(run_root losetup -f --show "$POOL_IMG")"

  run_root pvcreate -ff -y "$LOOP_DEV" >/dev/null
  run_root vgcreate "$VG_NAME" "$LOOP_DEV" >/dev/null
  if [ "$LVM_DEDICATED_METADATA" = "1" ]; then
    local meta_size="${LVM_META_SIZE:-8G}"
    run_root lvcreate -y -L "$meta_size" -n "$META_NAME" "$VG_NAME" >/dev/null
    run_root lvcreate -y -L "${LVM_POOL_SIZE:-180G}" -T "$VG_NAME/$POOL_NAME" --poolmetadatasize "$meta_size" >/dev/null
  else
    run_root lvcreate -y -L "${LVM_POOL_SIZE:-200G}" -T "$VG_NAME/$POOL_NAME" >/dev/null
  fi
  local main_lv
  main_lv="$(lv_name_for_branch main)"
  run_root lvcreate -y -V "$MAIN_SIZE" -T "$VG_NAME/$POOL_NAME" -n "$main_lv" >/dev/null

  local main_mount
  main_mount="$BRANCH_ROOT/main"
  mkdir -p "$main_mount"
  local main_path
  main_path="$(run_root lvs --noheadings -o lv_path "$VG_NAME/$main_lv" | awk '{print $1}')"
  run_root mkfs.ext4 -F "$main_path" >/dev/null
  run_root mount "$main_path" "$main_mount"
  mkdir -p "$main_mount/pgdata"

  upsert_branch "main" "" "$main_lv" "$main_mount"
  save_state
  init_parent_pgdata "$main_mount"
  echo "ok"
}

run-branching-runtime-bench.sh (skip backend on scenario failure)

function run_scenario_once() {
  local adapter="$1"
  local backend="$2"
  local scenario="$3"
  local seconds="$4"
  local run_dir="$5"

  "$adapter" cleanup >/dev/null 2>&1 || true
  if ! "$adapter" prepare-parent "$parent_size" >/dev/null 2>&1; then
    return 1
  fi

  # scenario execution
}

for backend in "${runtime_backends[@]}"; do
  adapter="$SCRIPT_DIR/backends/${backend}-adapter.sh"
  if [ ! -x "$adapter" ]; then
    echo "missing adapter: $adapter"
    continue
  fi
  if ! "$SCRIPT_DIR/backends/backend-contract.sh" "$adapter" check >/dev/null 2>&1; then
    echo "contract check failed: $backend"
    continue
  fi

  backend_failed=0
  for scenario in "${runtime_scenarios[@]}"; do
    for run_index in $(seq 0 "$repeats"); do
      is_warmup="0"
      seconds="$run_seconds"
      repeat_label="repeat-$run_index"
      if [ "$run_index" -eq 0 ]; then
        is_warmup="1"
        seconds="$warmup_seconds"
        repeat_label="warmup"
      fi
      run_dir="$result_dir/$backend/$scenario/$repeat_label"
      mkdir -p "$run_dir"
      if ! run_scenario_once "$adapter" "$backend" "$scenario" "$seconds" "$run_dir"; then
        echo "scenario failed: $backend $scenario $repeat_label"
        backend_failed=1
        break
      fi
      sleep "${RUN_COOLDOWN_SECONDS:-60}"
    done
    if [ "$backend_failed" -eq 1 ]; then
      break
    fi
  done

  "$adapter" cleanup >/dev/null 2>&1 || true
done

run-branching-research-all.sh (phase-1 override support)

phase1_result_override="${PHASE1_RESULT_DIR_OVERRIDE:-}"

if [ -n "$phase1_result_override" ]; then
  phase1_result="$phase1_result_override"
  log_line "phase1 override result=$phase1_result"
else
  run_batch phase1-ops run-branching-ops-bench.sh ""
  phase1_result="$LAST_BATCH_RESULT"
fi
phase1_gates="$phase1_result/ops-gates.csv"
phase1_summary="$phase1_result/ops-summary.csv"

if you want, i can post a second comment with full file contents for these scripts too.

[elitan]

additional note: concrete hang-fix sketch for run-branching-hetzner-batch.sh

this is the blocker we kept hitting. adding a patch-style snippet here so rerun can start clean.

# add near top (after remote_env/base_env setup)
remote_timeout_sec="${BENCHMARK_REMOTE_TIMEOUT_SEC:-43200}"
remote_marker="/root/benchmarks/.batch-done-$(date +%s)-$$"

# replace direct ssh execution with timeout + marker
remote_cmd="cd /root/benchmarks && . ./benchmark-config.env && env $base_env"
if [ -n "$remote_env" ]; then
  remote_cmd="cd /root/benchmarks && . ./benchmark-config.env && env $base_env $remote_env"
fi
remote_cmd="$remote_cmd ./$remote_script"
if [ -n "$remote_args" ]; then
  remote_cmd="$remote_cmd $remote_args"
fi

wrapped_remote_cmd="set -euo pipefail; ($remote_cmd); rc=\$?; echo \$rc > $remote_marker; exit \$rc"

set +e
timeout "$remote_timeout_sec" ssh -o StrictHostKeyChecking=accept-new root@"$server_ip" "$wrapped_remote_cmd"
ssh_code=$?
set -e

if [ "$ssh_code" -eq 124 ]; then
  echo "remote command timeout after ${remote_timeout_sec}s"
  exit 1
fi
if [ "$ssh_code" -ne 0 ]; then
  echo "remote command failed with code $ssh_code"
  exit "$ssh_code"
fi

if ! ssh -o StrictHostKeyChecking=accept-new root@"$server_ip" "test -f $remote_marker"; then
  echo "remote completion marker missing: $remote_marker"
  exit 1
fi

optional hardening:

# also add on ssh/scp invocations
-o ServerAliveInterval=15 -o ServerAliveCountMax=20 -o ConnectTimeout=10

strict rerun checklist (from scratch):

# 1) confirm no old benchmark vm
curl -sS -H "Authorization: Bearer $HETZNER_API_KEY" https://api.hetzner.cloud/v1/servers | jq -r '.servers[] | [.name,.status] | @tsv'

# 2) start full campaign from phase-1
./benchmarks/run-branching-research-all.sh

# 3) after finish, confirm cleanup again
curl -sS -H "Authorization: Bearer $HETZNER_API_KEY" https://api.hetzner.cloud/v1/servers | jq -r '.servers[] | [.name,.status] | @tsv'