model-cascade

Distill a repeated Jev classification task into a local model, on the fly — same answers, your hardware.

Cost-efficient coding-agent runtime: prefix-aware prompt-cache routing, poisoning-safe semantic cache, budgeted context compression, request batching/dedup, cascading model routing with DP-optimal policies, hard budget scheduling, span-level cost attribution, sandboxed tools, synthetic workload factory. NumPy core, CPU-only torch extra.

⚖️ Kadı — a calibrated judge in front of your frontier LLM. Routine decisions answered locally with a conformal-prediction guarantee (≤ α disagreement with the oracle); everything else escalated and learned from. Zero-dependency Python.

Distil an expensive LLM API call on a narrow task into a small local model: capture traffic, curate, LoRA fine-tune with MLX on Apple Silicon, evaluate against the teacher, and serve an OpenAI-compatible cascade that escalates low-confidence requests.

Confidence-gated LLM cascades that measure how often the cheap tier is wrong on the cases it answers alone.

Support inbox that routes each ticket to the cheapest model that can handle it: fine-tuned MiniLM first, LLM only when unsure.

Cost-aware routing across agentic pipelines. Empirical benchmarks of accuracy, cost, and dangerous rate.

Execution-based routing between a small and a large LLM for text-to-SQL. A 7B answers first, a classifier on its executed SQL decides when to escalate to a 32B. Matches the 32B on BIRD Mini-Dev at 30% escalation and 29% of the cost. Pre-registered experiments.

Studying Minimum Sufficient Inference: when objective execution evidence can stop LLM inference without sacrificing reliability.

AI Evals: evaluating and improving an AI customer support agent. Error analysis, a validated LLM judge, a CI quality gate that blocks regressions, drift monitoring with human review of every alert, targeted fixes compared on a score vs cost frontier, and cost reduction with a judge cascade..

Adaptive Cascade Tuning via Importance Sampling

NSGA-II search framework for CIFAR-10 big/little dynamic inference cascades under embedded memory constraints.

Distill a frontier-model task (gpt-oss-120b, banking77 intents) into a 1.5B QLoRA student served on vLLM, with a confidence router that escalates hard queries to the teacher. Measured: 90% of teacher accuracy alone, 99.7% routed at 28% escalation, break-even ~313K req/day. Spend-capped, Cloud Run + Terraform ready.

LLM serving router that prices every request in dollars and milliseconds. Semantic cache plus a confidence-gated cheap-to-GPT-4 cascade: 96.9% lower cost than always calling the strong model, within 1.6 accuracy points, median latency 450ms to sub-ms on repeated traffic. FastAPI, SQLite cost ledger, reproducible benchmark.

Route each LLM call to the cheapest model that will get it right. Verifier-gated cascades with an offline threshold optimizer and cost accounting that counts every verifier call at current list prices.

Model cascade with a verification gate: try the cheapest model first, escalate only when tests or a cross-vendor judge reject the result.

Ask the cheap model first. Pay for the strong one only when the cheap answer fails a check, under a budget, with a cost record for every call.

A programming language where decisions are the branches: typed, exhaustively matched decisions that escalate across a builtin model → Nodd → Jet. Also a TypeScript library.

A cost-aware decision layer for routing text requests: a cheap model first, an LLM only when confidence is low, a person when both are unsure. FastAPI service with a React dashboard, measured for accuracy, cost and latency.