Contextual Orchestrator

Stdlib Python lab for a single API that routes, delegates, verifies, and synthesizes work across a configurable pool of OpenAI-compatible model agents.

This is not a Sakana AI product or a reproduction of their trained models. It is a small implementation of the public architecture pattern: expose one model-like orchestration candidate while keeping the worker pool, routing, workflow, and verification logic behind it. contextual-orchestrator is the public control-plane model; it is not just an HTTP gateway.

Docker Compose

The canonical path is compose.yaml. It starts PostgreSQL, seeds separate admin and inference tokens into the encrypted KV from Compose secrets, and binds the gateway to loopback:

umask 077
mkdir -p .secrets
chmod 700 .secrets
printf '%s' 'replace-with-a-long-random-admin-token' > .secrets/admin-token
printf '%s' 'replace-with-a-long-random-inference-token' > .secrets/inference-token
chmod 600 .secrets/admin-token .secrets/inference-token
export INFERENCE_TOKEN="$(cat .secrets/inference-token)"
export CONTEXTUAL_ORCHESTRATOR_POSTGRES_PASSWORD='replace-with-a-database-password'
export CONTEXTUAL_ORCHESTRATOR_KV_PASSPHRASE='replace-with-an-encryption-passphrase'
docker compose up --build --wait
curl http://127.0.0.1:8000/healthz

Register provider keys separately with register-credential; do not put them in compose.yaml or the gateway runtime environment.

For orchestration with OpenAI Responses-native reasoning summaries, select orchestrator/auto or the fail-closed zero-cost pool orchestrator/free:

curl -N http://127.0.0.1:8000/v1/responses \
  -H "Authorization: Bearer $INFERENCE_TOKEN" -H 'Content-Type: application/json' \
  -d '{"model":"orchestrator/free","input":"Research and verify this","reasoning":{"summary":"auto"},"stream":true}'

Quick Start

python -m contextual_orchestrator "Summarize why model orchestration helps long coding tasks." \
  --agents examples/agents.mock.json

Run the OpenAI-compatible subset:

local_token="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
python -m contextual_orchestrator --serve --agents examples/agents.mock.json --port 8000 \
  --auth-token "$local_token"

Admin console:

http://127.0.0.1:8000/admin

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "authorization: Bearer $local_token" \
  -H 'content-type: application/json' \
  -d '{"model":"contextual-orchestrator","messages":[{"role":"user","content":"Analyze this code review task and verify the answer."}]}' | jq .

HTTP serving is hardened for local lab use:

• /admin, /admin/state, /api/v1/*, and /v1/chat/completions require a Bearer token. Use --admin-token-key and --inference-token-key to resolve split tokens from the KV, or --auth-token-key for one local token. Explicit --auth-token/split-token values are local-development escape hatches; --production and --allow-public-bind reject single-token mode and insecure admin-session cookies, and the CLI never reads auth secrets from environment variables.
• A production deployment that uses the ecosystem identity plane must inject a reviewed bearer_verifier into SecurityConfig to validate Keyverse-issued OIDC tokens (issuer, audience, signature, expiry, and scope). The core does not hand-roll JWT parsing or hold Keycloak admin credentials; a static bearer token is not a Keyverse integration.
• Binding to a non-loopback address requires --allow-public-bind; loopback addresses and localhost remain available for local development.
• JSON request bodies, chat message roles, orchestration modes, body sizes, request rate, and concurrent run counts are validated before orchestration runs.
• /healthz is a minimal unauthenticated process probe; use the administrator-authenticated /readyz endpoint for secret-free orchestration, sync-routing, and optional batch dependency status. Liveness stays available during optional dependency degradation.
• Full orchestration traces are not returned by default. Set include_orchestration_trace: true per standard Chat request or start with --expose-trace-by-default when the caller is trusted. Requests that also use tools or response_format still fail with unsupported_trace_disclosure; remove the trace flag for structured or tool requests.
• State is in-memory by default. Pass --state-db PATH (or CONTEXTUAL_ORCHESTRATOR_STATE_DB) to persist workflow runs, evaluation runs, audit, and analytics to a stdlib sqlite file so they survive a restart; without it, behavior is unchanged.
• Response caching is off by default. Pass --cache-ttl SECONDS to serve identical requests (same messages + mode) from an in-memory TTL+LRU cache and skip the provider calls; 0 disables it.
• ModelClient.batch_chat(agent, {custom_id: messages}) runs many requests through the provider's Batch API (async, 24h completion window, typically ~50% cheaper) — suited to evaluation/benchmark workloads, not latency-sensitive chat. The mock path answers synchronously.

Use real workers by replacing mock:// agents with OpenAI-compatible endpoints. Provider secrets are resolved from a KV credential registry via get_credential, never from os.getenv at request time (see docs/kv-credentials.md):

{
  "agents": [
    {
      "id": "coding_agent",
      "model": "gpt-5.5",
      "base_url": "https://api.openai.com/v1",
      "credential_key": "OPENAI_API_KEY",
      "tags": ["coding", "debugging", "reasoning"]
    }
  ]
}

For a local mlx-lm OpenAI-compatible server, use the explicit mlx:// scheme. It is loopback-only, does not require a credential, and is translated to HTTP only after the loopback check:

{
  "agents": [
    {
      "id": "local_fast_agent",
      "model": "mlx-community/llama-3.2-3b-instruct-4bit",
      "base_url": "mlx://127.0.0.1:8080/v1",
      "provider_name": "mlx-lm",
      "tags": ["reasoning", "coding", "verification"]
    }
  ]
}

The full local candidate registry is examples/agents.local.json. It contains the public contextual-orchestrator candidate, discovered MLX worker models, and every discovered llama.cpp/LM Studio candidate. Discovery does not decide governance state: seed candidates are enabled by default, while disabled is reserved for an explicit operator/admin quarantine or a persisted removal tombstone. The contextual-orchestrator record is excluded from internal roles because this implementation has no bounded recursive self-call protocol; that is a routing safety constraint, not a disabled candidate. The registry is explicit; runtime discovery does not silently change the pool.

Run an evaluation against that server with --temperature 0 for repeatable judging. For reasoning-capable mlx models, pass --chat-template-args '{"enable_thinking":false}' when a short structured judge response is required. --local-concurrency N enables bounded concurrent local batch requests (1..64; the current measured starting point for this server is 8); when serving HTTP, set --max-concurrent-runs N explicitly as well if the measured batch concurrency exceeds the secure default of 8. Keep interactive route/conduct requests on the default sequential path.

Model-based conduct verification requires fast-mlsirm in the same runtime and fails closed when it is absent or broken; fast-mlsirm sends its judge completion through this contextual-orchestrator gateway, so no direct provider fallback is used. “Same runtime” means that the exact interpreter used for the live run can import both packages: install both checkouts into one environment (prefer editable installs), or expose both source roots with PYTHONPATH during a source run. Before a live judge benchmark, run python -m contextual_orchestrator check-fast-mlsirm with that exact interpreter. It prints the interpreter, package version, transitive-import status, and contextual contract check, and exits nonzero on a missing dependency or contract mismatch. Do not run the preflight in one virtual environment and the judge in another. See ADR 0001.

The agent pool is manageable at runtime: POST/PATCH/DELETE on /api/v1/agent_pools/default/worker_agents[/{id}] add, govern, and remove model-group members. Pass --agents-db PATH (or CONTEXTUAL_ORCHESTRATOR_AGENTS_DB) to persist those changes to a stdlib sqlite file — stored changes overlay the seed agents file at startup, and removals write disabled tombstones so they survive restarts; without it the pool is in-memory as before. Beyond the local MLX/llama.cpp discovery above, python -m contextual_orchestrator discover-models [--agents-db PATH] discovers models from remote providers (OpenAI, OpenRouter, NVIDIA NIM ×2 keys, Bytez, and an allowlisted OpenAI-compatible gateway) for any subset of their KV-registered credentials, and can persist them into the same --agents-db sqlite file, added disabled by default. See docs/kv-credentials.md for the credential-name table and cost-based auto-selection.

Seed the credential into the KV once at bootstrap:

echo "$OPENAI_API_KEY" | python -m contextual_orchestrator register-credential --name OPENAI_API_KEY --value-stdin

For a persistent KV-backed server token, seed a credential such as CONTEXTUAL_ORCHESTRATOR_TOKEN and start with --auth-token-key CONTEXTUAL_ORCHESTRATOR_TOKEN. The in-memory credential backend is process-local and is suitable only for tests; production auth registration and OIDC client secrets belong to the deployment/KV boundary.

Non-mock providers must use https:// URLs and a resolvable KV credential — a non-mock agent whose credential is missing raises NotConfigured rather than falling back to an environment variable. The runtime blocks loopback, private, link-local, multicast, and reserved provider addresses before sending a key. Pass --allowed-provider-host HOST once per approved gateway when an explicit host allowlist is required; it is bound at client construction and is not changed by request-time environment variables. External calls use a timeout and default output token cap.

The legacy api_key_env field is still accepted for back-compat, but its value is now treated as the credential name in the KV, not as an environment variable to read. This supersedes the old api_key_env env pattern.

Architecture

One public interface:

• contextual-orchestrator is the model-like control-plane candidate exposed to callers. /v1/models lists it first, followed by every configured worker candidate, including disabled candidates with their status.
• /v1/chat/completions accepts normal chat messages, and "stream": true returns an OpenAI-compatible text/event-stream of chat.completion.chunk deltas terminated by data: [DONE]. stream_options.include_usage=true is accepted for ordinary chat streams and emits a usage-only chunk after the terminal stop chunk. Valid provider counts carry usage_source: reported and usage_measurement_status: measured; missing or malformed counts carry usage: null and usage_measurement_status: unavailable. Single-agent tools passthrough follows the same rule and never reconstructs tool or multimodal framing. response_format-only structured passthrough (conduct mode) still rejects the combination before provider execution when workflow-level usage is unavailable. In route mode the worker's tokens are streamed live as they arrive from the provider (real token streaming); in conduct mode the multi-step answer is produced then framed as deltas (a workflow can't honestly token-stream a synthesizer that hasn't run yet).
• TaskOrchestrator.complete() decides whether to route to one worker or run a short workflow.
• TaskOrchestrator.compare_to_baseline(prompts, mode) (CLI --eval PROMPT...) measures the orchestration engine against a single-worker baseline — per-prompt and aggregate latency plus a structural coverage delta (contributing steps + verifier-pass presence). It is a measured tradeoff report, not a human-quality claim.
• Responses include orchestration mode metadata, and trusted callers can request the full trace for audit.
• /admin exposes an operator console for agent pool, policy, trace, and audit review.
• The admin console can use Clearfolio as its document viewer: pass --clearfolio-url URL (or CONTEXTUAL_ORCHESTRATOR_CLEARFOLIO_URL) and the Integrations view gains a Document Viewer card (open viewer / deep-link {url}/viewer/{docId}). Default: disabled, console unchanged.
• /api/v1/provider_readiness/latest reports provider liveness separately from an explicit chat readiness probe; ?refresh=true re-probes instead of returning the cached result.
• /api/v1/analytics_snapshots/latest returns source-backed local KPI definitions (trace completeness, policy-safe run rate, successful chat requests, and related event-derived counts) from in-memory runtime state, localized via the same locale bundles as the admin console.
• /api/v1/spend_analytics/latest exposes per-model token and cost spend aggregated from workflow runs. Valid provider usage is authoritative; declared model IDs may use the packaged Rust tokenizer for exact raw textual output. Prompt framing, tools, multimodal input, unknown tokenizers, and missing native code remain unavailable. Cost is computed only when every required count and operator-supplied price is available. See Observability & spend.
• /api/v1/sales_readiness/latest exposes a local enterprise-pilot readiness gate for API compatibility, operator evidence, workflow traces, evaluation replay, security posture, analytics truthfulness, locale parity, and provider egress safety. It is process-local evidence, not a production compliance certificate.
• /api/v1/commercial_readiness/latest exposes a KRW 2,000,000,000 commercial due-diligence readiness gate. It is a buyer-review evidence snapshot, not a valuation guarantee or purchase commitment.
• /api/v1/commercial_evidence_manifests/latest shows the evidence gaps to resolve before commercial due diligence. The former /api/v1/buyer_evidence_manifests/latest route remains a deprecated compatibility alias.
• /api/v1/commercial_handoff_bundles/latest shows the handoff evidence and remaining commercial follow-ups. The former /api/v1/buyer_handoff_bundles/latest route remains a deprecated compatibility alias.
• /api/v1/saleability_decisions/latest exposes the final KRW 2,000,000,000 saleability decision gate with concrete blockers, warning conditions, and review-process non-blocker policy.
• /api/v1/commercial_evidence_exports/latest exposes the portable commercial evidence export across saleability, runtime reports, buyer documents, Figma artifacts, verification commands, review-process policy, packaging decision, and external evidence gaps.
• /api/v1/commercial_acceptance_checks/latest exposes the buyer acceptance check across evidence export, runtime endpoint chain, buyer packet, admin surface, verification, Figma, review-process policy, packaging decision, and external evidence gaps.
• /api/v1/commercial_buyer_acceptance_workflows/latest exposes the buyer acceptance workflow across owner-scoped runbook steps, Go/Warning/No-Go rules, runtime evidence, Figma artifacts, analytics truthfulness, review-process policy, and packaging decision.
• /api/v1/commercial_release_candidates/latest exposes the local commercial release-candidate manifest across acceptance, runtime endpoints, repository distribution packet, security metadata, admin surface, verification, Figma, review-process policy, packaging decision, and external release gaps.
• /api/v1/commercial_gap_registers/latest exposes the commercial gap register that turns release-candidate external gaps into owner, source, required-input, and status rows for buyer due diligence.
• /api/v1/commercial_procurement_readiness/latest exposes the commercial procurement readiness gate across license, rights, security metadata, distribution packet, admin evidence, production support/SLO input, buyer legal/ROI/procurement input, review-process policy, and packaging decision.
• /api/v1/commercial_contract_readiness/latest exposes the commercial contract readiness gate across support/SLO terms, security/privacy terms, audit/export obligations, license/commercial rights, buyer order-form inputs, review-process policy, and packaging decision.
• /api/v1/commercial_onboarding_readiness/latest exposes the commercial onboarding readiness gate that turns production support/SLO and buyer-specific input warnings into paid-onboarding owners, actions, and exit criteria.
• /api/v1/commercial_operations_readiness/latest exposes the commercial operations readiness gate that turns production telemetry, incident/rollback, backup/recovery, and SLO evidence gaps into operations handoff owners, actions, and exit criteria.
• /api/v1/commercial_security_attestations/latest exposes the commercial security attestation gate that separates repo-local security evidence from external attestation, hosted scan, and buyer privacy/DPA gaps.
• /api/v1/commercial_value_readiness/latest exposes the commercial value readiness gate that separates repo-local measured value evidence from buyer-specific ROI, reference proof, budget-owner, and payback-input gaps.
• /api/v1/commercial_close_readiness/latest exposes the commercial close readiness gate that separates repo-local sellable product evidence from buyer signatures, DPA/security acceptance, budget/PO, and go-live authorization gaps.
• /api/v1/commercial_go_to_market_readiness/latest exposes the commercial go-to-market readiness index that ties close, value, security, evidence export, buyer handoff, saleability, admin evidence, analytics truthfulness, Figma artifacts, review-process policy, and packaging decision into one buyer/stakeholder review packet.
• /api/v1/commercial_launch_readiness/latest exposes the commercial launch readiness gate that packages GTM, runtime, acceptance, operator, admin, analytics, Figma, review-process, and packaging evidence while keeping buyer environment, production telemetry, and signature inputs as explicit warnings.
• /api/v1/commercial_completion_scorecards/latest exposes the runtime commercial completion scorecard for the KRW 2,000,000,000 program-completion standard across Product Design, Figma, Superpowers, Ponytail, Data Analytics, runtime, verification, review-policy, packaging, and external follow-up evidence.
• /api/v1/commercial_demo_scenarios/latest exposes the KRW 2,000,000,000 commercial demo scenarios packet across compatible API smoke, workflow trace, access-list evidence, evaluation replay, admin readiness, metric truthfulness, Figma review, buyer acceptance, review-process policy, and packaging decision.
• /api/v1/commercial_proposal_packets/latest exposes the KRW 2,000,000,000 commercial proposal packet across completion, demo, acceptance, value, security, contract, onboarding, operations, analytics truthfulness, Figma review, review-process policy, packaging decision, and buyer-specific follow-ups.
• /api/v1/commercial_purchase_approval_packets/latest exposes the KRW 2,000,000,000 commercial purchase approval packet across proposal, close, procurement, contract, value, security, onboarding, operations, analytics truthfulness, Figma review, review-process policy, packaging decision, and buyer signature/budget authority follow-ups.
• /api/v1/commercial_due_diligence_rooms/latest exposes the KRW 2,000,000,000 commercial due diligence room across purchase approval, runtime API evidence, admin trace/access evidence, security, commercial terms, value analytics, implementation readiness, Figma review, review-process policy, packaging decision, and buyer/external missing artifacts.
• /api/v1/commercial_investment_committee_memos/latest exposes the KRW 2,000,000,000 commercial investment committee memo across due diligence, purchase approval, financial case, risk/security, commercial terms, implementation readiness, Figma review, review-process policy, packaging decision, and buyer/external approval conditions.

One fused orchestration loop:

• Fast path: one worker is selected for simple or latency-sensitive requests.
• Deep path: a natural-language workflow is built with planner, worker, verifier, and synthesizer steps.
• Each step has an access list, so workers see only the prior outputs intentionally exposed to them.
• Agent definitions are data, so provider preference, exclusions, privacy constraints, and mock testing do not require code changes.
• Provider calls are resilient: transient failures (timeouts, 429, 5xx) retry with full-jitter exponential backoff, while caller errors (4xx) fail fast; a provider's explicit tool-description size rejection is treated as a provider limit and can fail over. If an agent still fails, the request fails over to the next capability-matched agent in the pool, and a per-agent circuit breaker skips a persistently failing provider until it cools down. Failover is recorded in the trace (served_agent_id, failover_from).

See docs/architecture.md for the source-backed analysis.

Observability & spend

Local spend observability, aggregated from in-memory workflow runs. It is honest by construction: counts are authoritative or explicitly unavailable, and cost is reported only when its required counts and prices are available.

curl -s http://127.0.0.1:8000/api/v1/spend_analytics/latest \
  -H "authorization: Bearer $local_token" | jq '.totals, .by_model, .budget'

• Tokens. by_model[].output_tokens uses provider-reported completion/output tokens first. For exact full model IDs declared by ADR 0006, a missing output count may use the packaged Rust tokenizer over raw textual output only. Rows carry usage_source: reported | tokenizer | mixed | unavailable; unavailable rows return output_tokens: null. Prompt tokens are provider-reported or null because chat framing is not reconstructed.

• Cost. Supply a price table to turn authoritative output tokens into money — TaskOrchestrator(price_per_million={"gpt-5.5": 10.0}) (USD per 1M output tokens). Models without a price appear under unpriced_models; cost_usd remains null when a price or required token count is unavailable. No prices or token counts are assumed.

• Budget cap. Set an operator cap to refuse runaway spend (default: no cap):

  python -m contextual_orchestrator --serve --agents examples/agents.mock.json \
    --budget-max-output-tokens 2000000 --budget-max-cost-usd 50

  Or in code: TaskOrchestrator(budget_max_output_tokens=..., budget_max_cost_usd=...). Once spend reaches a cap, the next run is refused — run() raises BudgetExceededError and /v1/chat/completions returns HTTP 429 budget_exceeded. An enabled budget also fails closed when a required count or price is unavailable. Current state is in spend_analytics()["budget"] (enabled, limits, nullable spent_*/remaining_*, measurement_status, enforcement_status, exceeded). Cost caps require a complete price table; token caps require authoritative output counts.

• Admin. The /admin Observability view renders the totals and the per-model table (unpriced models show an unpriced chip).

These are process-local measured signals for a stdlib lab, not a billing system or production compliance data.

Cost review + routing hub

The orchestrator is the single control point for LLM cost review and sync-vs-batch routing (a LiteLLM-plus scope: cost optimiser + upstream load balancing + batch routing). All config — prices, thresholds, batch endpoints — is read from a KV config store, never os.getenv.

• Usage + cost ledger. Every completion, sync and batch, builds a prompt-safe usage record with generated IDs, token counts, cost, provider, model, channel, route mode, and attribution dimensions. Raw prompt and answer text are not part of the usage record or telemetry event. The default in-memory ledger keeps local reports available; external persistence/export should be wired through the OpenTelemetry-shaped usage event sink or the bounded NonBlockingLedgerStore, so store failures are observable as usage export health without failing completions. Cost is attributable on seven first-class dimensions catalogued in cost_attribution_dimensions: account, service, upstream API/provider, model name, team, group, company. Token counts reuse pg-llm-batch's pg_tiktoken counter when a Postgres DSN is configured. Valid provider usage is authoritative; missing chat framing, tool, multimodal, or unknown-tokenizer counts remain explicitly unavailable instead of falling back to a deterministic heuristic.
• Canonical Billing export. Install the published metering_billing producer SDK, create its durable outbox, and pass CanonicalUsageRecordSink(event_builder=build_contextual_usage_event, enqueue=outbox.enqueue, identity=...) as CostLedger(usage_sink=...). The sink runs only after the local ledger accepts a new record, so failed, dropped, and duplicate writes do not become billing-only events. For a caller-owned SQLite transaction, export is deferred until flush() observes the caller's commit; rollback therefore emits no billing-only event. A non-blocking SQL store writes such a billing-backed append synchronously while the transaction is open and defers export until flush() confirms the commit, so a background worker cannot race the caller's transaction outcome. It exports token counts and bounded provider/model/workflow metadata; prompts, answers, and computed prices never enter the event.
• Reporting. GET /api/v1/cost_reports/rollup?dimension=team&start=&end= rolls up cost + tokens by any dimension over any time window; GET /api/v1/llm_usage_records lists raw ledger rows; GET /api/v1/cost_attribution_dimensions lists the dimension catalog.
• Routing. RoutingPolicy decides sync vs batch from request hints ({"routing": {"latency_tolerant": true}} on /v1/chat/completions) plus KV thresholds. Interactive requests stay on the fast sync path; latency-tolerant or bulk requests are dispatched to a batch backend.
• Batch routing to pg-llm-batch. The production batch backend is an injected pg-llm-batch OpenAI-compatible Batch API client (submit JSONL -> poll -> retrieve). A local in-process backend preserves the mock/standalone path with no external service or repository split. Submit via POST /api/v1/batch_routing_jobs, poll GET /api/v1/batch_routing_jobs/{id}, retrieve POST /api/v1/batch_routing_jobs/{id}/results (which records usage + cost).
• Batch embeddings. Bulk, latency-tolerant embedding work (e.g. naruon's email-import backfill) submits to POST /v1/batch/embeddings ({model, input|inputs:[...], endpoint, metadata|attribution}) and polls GET /v1/batch/embeddings/{batch_id}. The response is {batch_id, status, embeddings:[{index, embedding}], cost_micro_usd, token_counts, total_tokens, part_count, input_part_counts, map_reduce}. Before the provider call, the coordinator maps oversized inputs into token-budgeted embedding parts (routing.embedding_max_tokens_per_request, default 280,000; routing.embedding_max_chars_per_part, default 240,000) and reduces part vectors with a token-weighted average, so Azure/LiteLLM over-limit embedding requests are split internally instead of surfacing as caller errors. It routes through the same RoutingPolicy/cost optimiser and pg-llm-batch embeddings backend (local in-process backend standalone), and records one usage-ledger row per original vector with the full attribution dimensions (service, team, group, company, provider) carried in metadata.
• Health. GET /healthz is an unauthenticated liveness probe that returns only service identity and process status; it never discloses worker topology, backend names, usage volume, or upstream readiness. Admins can use GET /api/v1/provider_readiness/latest?refresh=true for one bounded, non-retrying chat probe per enabled worker.
• Standalone + optional pg-llm-batch integration. The hub runs standalone with the in-memory config store and local batch backend; wiring a Postgres DSN and an installed/deployed pg_llm_batch client activates the KV/secret stores, pg_tiktoken counting, and the production batch backend without adding a repository split here.

Grounding papers (LLM cost, routing, load balancing, evaluation) live in docs/papers with citations.

NIM cost-quality benchmark (optional harness)

Evidence-grade benchmark of the routing policies against a dynamically discovered NVIDIA NIM catalog. It probes chat, completions, Responses, embeddings, image/video/audio understanding, transcription, and speech; compares direct, route-once, and bounded-conduct cells under one equal total-token and call budget; records paired uncertainty and Pareto frontiers; and keeps reviewed actual endpoint-access evidence separate from optional hypothetical paid rates. The bundled manifest is smoke-sized and reports evidence_review_required only after its paired cells complete; routing_recommendation remains null and no benchmark artifact automatically changes production routing.

The adapter is lazy and optional: ordinary import contextual_orchestrator does not import or mutate it. Deterministic --dry-run receives no network access or NVIDIA secret. Live execution resolves NVIDIA_NIM_API_KEY from the credential registry, pins HTTPS connections to validation-time public addresses, rejects redirects and proxy routing, and fails closed on missing/expired evidence. See docs/nim_benchmark.md and the engineering decision record.

python -m contextual_orchestrator nim-benchmark --dry-run \
  --pricing-scenario examples/nim_pricing_scenario.json \
  --output-dir benchmark_artifacts

Design Artifacts

• Library research
• Product planning
• Screen design
• User stories
• REST API design
• Code conventions
• Database conventions
• i18n design
• Plugin-driven design brief
• Plugin visual directions
• Analytics spec
• Commercial readiness standard
• Commercial buyer diligence packet
• Commercial buyer acceptance runbook
• Commercial buyer evidence manifest
• Commercial buyer handoff bundle
• Commercial saleability decision
• Commercial evidence export
• Commercial acceptance check
• Commercial release candidate
• Commercial gap register
• Commercial procurement readiness
• Commercial contract readiness
• Commercial onboarding readiness
• Commercial operations readiness
• Commercial security attestation
• Commercial value readiness
• Commercial close readiness
• Commercial go-to-market readiness
• Commercial launch readiness
• Commercial completion scorecard
• Commercial demo scenarios
• Commercial proposal packet
• Commercial purchase approval packet
• Commercial due diligence room
• Commercial investment committee memo
• Commercial plugin operating model
• Figma artifacts
• Fuzzing
• Product and technical gap baseline
• Plugin-driven implementation plan
• Commercial plugin readiness plan

Check

Run the full suite with the same hash-locked pytest toolchain as CI:

make test

For the individual smoke checks, use the existing runtime lock:

python -m pip install --require-hashes -r requirements.lock
python -m pip install --no-deps -e .
python tests/test_self_check.py
python tests/test_paper_contracts.py
python -m pytest -q tests/test_reasoning_effort_profile.py
python tests/test_admin_contract.py
python tests/test_conventions.py
python tests/test_api_contract.py
python tests/test_nim_benchmark.py
python tests/test_security_hardening.py
python tests/test_chat_model_capability_isolation.py
python tests/test_chat_transport_role_separation.py
python tests/test_chat_capability_unknown_identifiers.py
python tests/test_chat_passthrough_capability_isolation.py
python tests/test_discovery_bootstrap_selection.py
python tests/test_chat_capability.py
python tests/test_review_gateway.py
python tests/test_provider_bootstrap.py
python tests/test_provider_bootstrap_secret_normalization.py
python tests/test_provider_catalog_bootstrap.py
python tests/test_provider_catalog_credential_promotion.py
python tests/test_provider_catalog_store.py
python tests/test_tool_execution_fallback.py
python tests/test_repository_security_metadata.py
python tests/test_product_planning_contract.py
python tests/test_plugin_driven_artifacts.py
python tests/test_analytics_runtime.py
python tests/test_cost_ledger.py
python tests/test_sales_readiness.py
python tests/test_buyer_evidence_manifest.py
python tests/test_buyer_handoff_bundle.py
python tests/test_saleability_decision.py
python tests/test_commercial_evidence_export.py
python tests/test_commercial_acceptance_check.py
python tests/test_release_authorization.py
python tests/test_release_authority_snapshot.py
python tests/test_commercial_buyer_acceptance_workflow.py
python tests/test_commercial_release_candidate.py
python tests/test_commercial_gap_register.py
python tests/test_commercial_procurement_readiness.py
python tests/test_commercial_contract_readiness.py
python tests/test_commercial_onboarding_readiness.py
python tests/test_commercial_operations_readiness.py
python tests/test_commercial_security_attestation.py
python tests/test_commercial_value_readiness.py
python tests/test_commercial_close_readiness.py
python tests/test_commercial_go_to_market_readiness.py
python tests/test_commercial_launch_readiness.py
python tests/test_commercial_completion_scorecard.py
python tests/test_commercial_demo_scenarios.py
python tests/test_commercial_proposal_packet.py
python tests/test_commercial_purchase_approval_packet.py
python tests/test_commercial_due_diligence_room.py
python tests/test_commercial_investment_committee_memo.py