Skip to content

README.md

Latest commit

 

History

History
380 lines (288 loc) · 13 KB

README.md

File metadata and controls

380 lines (288 loc) · 13 KB
Raw

Test and Demo Scripts

test/ contains local entrypoints, smoke scripts, and historical utility tests for CacheRoute development. The main demo_*.py entrypoints add the repository root to Python's module search path, so the documented commands can be launched directly from the test directory without manually setting PYTHONPATH.

Install the complete application and development dependency set from the repository root before running the commands in this document:

python3 -m pip install -r requirements.txt
python3 -m pip check

If an older checkout reports ModuleNotFoundError: No module named 'core', update the checkout or temporarily run:

export PYTHONPATH="$(cd .. && pwd)${PYTHONPATH:+:$PYTHONPATH}"

The default single-machine demo ports are:

Component Service plane Control plane / auxiliary
Scheduler 7001 7002
Proxy 8001 8002
Proxy UI - 8202
Client UI - 7071
Instance 9001 9002
KDN Server 9101 -
Resource Agent 9201 -
Instance Resource Dashboard - 9202

Browser frontend URLs

Component Default URL Started by Notes
Proxy UI http://127.0.0.1:8202 demo_proxy.py Enabled by default. Use --no-proxy-ui to disable.
Client UI http://127.0.0.1:7071 demo_client.py --with-ui Sends OpenAI-compatible requests to Scheduler.
Instance Resource Dashboard http://127.0.0.1:9202 instance/resource_dashboard/dashboard_server.py Browser fallback for the Rust Resource Agent.
Scheduler UI TBD TBD Planned.
KDN Server UI TBD TBD Planned.

Main demo files

File Purpose Typical usage
demo_scheduler.py Starts Scheduler service plane and control plane. Use it when validating full Client -> Scheduler -> Proxy routing. python3 demo_scheduler.py --cacheroute
demo_proxy.py Starts Proxy service plane, Proxy control plane, and the browser Proxy UI by default. It registers to Scheduler if available, and accepts Instance registration/resource reports on 8002. python3 demo_proxy.py --strategy round_robin --injection-strategy iws
demo_instance.py Starts an Instance service. By default it registers to Proxy, starts or reuses the Rust Resource Agent, reports resource snapshots after registration, and cleans up demo-owned agent processes on shutdown. python3 demo_instance.py --host 127.0.0.1 --port 9001 --proxy-cp-url http://127.0.0.1:8002
demo_kdn.py Starts the KDN Server for text/KVCache metadata query and registration. python3 demo_kdn.py
demo_client.py Sends demo requests through CacheRoute. Can be used with or without browser UI. python3 demo_client.py --with-ui
demo_run.py Historical helper for running a demo flow. Check the script body before relying on it for new experiments. python3 demo_run.py
demo_embedding.py Local embedding / retrieval validation helper. python3 demo_embedding.py
demo_request_handle.py Request parsing and request-handle validation helper. python3 demo_request_handle.py
demo_resource_monitor_e2e.py Starts demo Proxy and demo Instance, waits for resource reports, terminates Instance, and checks that the demo-owned Rust Resource Agent is cleaned up. python3 demo_resource_monitor_e2e.py

Utility and regression scripts

File Purpose
request_handle.py Helper module for request parsing/handling experiments.
test.py Legacy scratch/test entrypoint. Inspect before use.
test_kb_kid.py Knowledge-base / knowledge-id validation.
test_injector_reuse.py Injector reuse validation.
test_kv_injector_reuse.py External-service KVCache reuse script. It requires a reachable vLLM-compatible endpoint at 127.0.0.1:8000 and is not an isolated CPU-only unit test.
Injection_method_comparison.py Compares injection methods for local experiments.
Prefill_calculation.py Prefill-time calculation helper.
quick_start_docker.sh Convenience script for container-oriented startup.

Focused Instance capability validation

The Instance capability contract covers deterministic fingerprints, structured compatibility results, capability-aware registration, and heartbeat recovery while preserving legacy payloads.

Run the focused CPU-only suite from the repository root:

python3 -m compileall core instance proxy

pytest -q test/test_instance_capability.py
pytest -q test/test_instance_capability_registration.py
pytest -q proxy/resource/test_instance_pool_gpu.py

pytest -q \
  test/test_instance_capability.py \
  test/test_instance_capability_registration.py \
  proxy/resource/test_instance_pool_gpu.py

The focused tests verify:

  • canonical and deterministic sha256: capability fingerprints;
  • model, tokenizer, KV layout, dtype, and parallelism mismatch reporting;
  • legacy registration and instance-ID-only heartbeats;
  • server-authoritative fingerprint calculation;
  • fresh registration replacement versus heartbeat omission preservation;
  • fingerprint-only heartbeat match, mismatch, and full-capability recovery;
  • InstancePool resource behavior.

Validate the previously public imports with:

python3 - <<'PY'
from core import (
    MLAmodel,
    Request,
    Prompt,
    Service,
    Task,
    TokenizerRegistry,
    forward_request,
)
from instance import (
    instance,
    mock_chat_stream,
    mock_chat_completion,
    mock_text_completion,
)
from proxy import proxy

assert all(
    item is not None
    for item in (
        MLAmodel,
        Request,
        Prompt,
        Service,
        Task,
        TokenizerRegistry,
        forward_request,
        instance,
        mock_chat_stream,
        mock_chat_completion,
        mock_text_completion,
        proxy,
    )
)
print("public import compatibility: passed")
PY

For a registration smoke test, start demo_proxy.py and launch demo_instance.py with explicit capability values:

INSTANCE_MODEL_ID=test-model \
INSTANCE_TOKENIZER_ID=test-tokenizer \
INSTANCE_KV_LAYOUT=paged \
INSTANCE_KV_SCHEMA_VERSION=1 \
INSTANCE_KV_DTYPE=fp16 \
INSTANCE_KV_BLOCK_SIZE=16 \
INSTANCE_TENSOR_PARALLEL_SIZE=1 \
INSTANCE_PIPELINE_PARALLEL_SIZE=1 \
INSTANCE_DATA_PARALLEL_SIZE=1 \
python3 test/demo_instance.py \
  --host 127.0.0.1 \
  --port 9001 \
  --proxy-cp-url http://127.0.0.1:8002 \
  --no-resource-monitor

Then inspect the accepted capability object and server-computed fingerprint:

curl -sS \
  "http://127.0.0.1:8002/v1/instance/list?include_dead=true" \
  | python3 -m json.tool

The result should expose the model, tokenizer, KV layout/dtype/block size, parallelism values, detected runtime versions when available, and a capability_fingerprint beginning with sha256:.

External-service collection note

test/test_kv_injector_reuse.py performs requests to http://127.0.0.1:8000/v1/chat/completions at module import time. Generic pytest -q collection can therefore fail with ConnectionError when the external vLLM-compatible endpoint is not running. This is an environment-dependent historical integration script rather than a failure of the capability unit tests.

For a CPU-only local run that does not provide the external endpoint, exclude that script explicitly:

pytest -q --ignore=test/test_kv_injector_reuse.py

Run the script directly only after the endpoint and its configured model are ready:

python3 test/test_kv_injector_reuse.py

Minimal Proxy + Instance resource-monitor demo

Terminal 1:

cd test
python3 demo_proxy.py \
  --host 127.0.0.1 \
  --port 8001 \
  --strategy round_robin \
  --injection-strategy iws

demo_proxy.py prints the Proxy UI URL when the UI starts successfully:

[demo_proxy] Proxy UI available at: http://127.0.0.1:8202

Terminal 2:

cd test
python3 demo_instance.py \
  --host 127.0.0.1 \
  --port 9001 \
  --proxy-cp-url http://127.0.0.1:8002

demo_instance.py enables resource monitoring by default. It will:

  1. register the Instance to Proxy;
  2. start or reuse the Rust Resource Agent;
  3. wait for http://127.0.0.1:9201/healthz;
  4. periodically report snapshots to http://127.0.0.1:8002/v1/instance/resource_snapshot;
  5. kill only the Resource Agent process group it started when Instance exits.

Inspect resource status with curl:

curl -sS "http://127.0.0.1:8002/debug/instance_resources" | python3 -m json.tool
curl -sS "http://127.0.0.1:8002/v1/instance/list?include_dead=true" | python3 -m json.tool

Or open the Proxy UI:

http://127.0.0.1:8202

Disable resource monitoring:

python3 demo_instance.py --no-resource-monitor

Use a non-default Resource Agent port:

python3 demo_instance.py \
  --resource-agent-listen 127.0.0.1:19201 \
  --resource-agent-url http://127.0.0.1:19201

Client browser UI

Start the Client UI:

cd test
python3 demo_client.py --with-ui

Open:

http://127.0.0.1:7071/ui/client

Override the listen address or Scheduler URL when needed:

python3 demo_client.py --with-ui \
  --ui-host 0.0.0.0 \
  --ui-port 7071 \
  --scheduler-url http://127.0.0.1:7001/v1/chat/completions

Resource-monitor e2e smoke script

Run from the repository root:

python3 test/demo_resource_monitor_e2e.py \
  --agent-listen 127.0.0.1:19201 \
  --agent-url http://127.0.0.1:19201

The non-default 19201 port avoids false failures if a separate Resource Agent is already running on 9201.

The script is intentionally a practical smoke validation rather than a pytest test. It may skip or fail on machines without cargo, uvicorn, or available ports.

Full local path

A typical local end-to-end flow uses separate terminals:

# Terminal 1: KDN
cd test
python3 demo_kdn.py

# Terminal 2: Scheduler
cd test
python3 demo_scheduler.py --cacheroute

# Terminal 3: Proxy
cd test
python3 demo_proxy.py --strategy round_robin --injection-strategy iws
# Open http://127.0.0.1:8202 for the Proxy UI.

# Terminal 4: Instance
cd test
python3 demo_instance.py --host 127.0.0.1 --port 9001 --proxy-cp-url http://127.0.0.1:8002

# Terminal 5: Client
cd test
python3 demo_client.py --with-ui
# Open http://127.0.0.1:7071/ui/client for the Client UI.

For focused Proxy/Instance development, Scheduler and KDN are optional unless the specific feature under test requires them.

Cleanup tips

Check ports:

ss -ltnp | grep -E "7001|7002|7071|8001|8002|8202|9001|9002|9101|9201|9202" || true

Find stale demo processes:

ps -ef | grep -E "demo_scheduler|demo_proxy|demo_instance|demo_kdn|demo_client|proxy_ui|resource_agent|cargo" | grep -v grep

If old external processes are heartbeating to Proxy with unknown IDs, inspect their environment:

for p in /proc/[0-9]*; do
  pid=${p##*/}
  env=$(tr '\0' '\n' < "$p/environ" 2>/dev/null | grep -E "INSTANCE_ID|PROXY_CP_URL" || true)
  if [ -n "$env" ]; then
    echo "PID=$pid CMD=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
    echo "$env"
    echo
  fi
done

Integrated Instance Resource Dashboard

demo_instance.py can optionally start the browser Resource Dashboard as part of the same Instance process:

python3 test/demo_instance.py --ui

With default settings, the Dashboard listens on 0.0.0.0:9202 and the usable local URL printed by the demo is:

http://127.0.0.1:9202

The listen address 0.0.0.0 means the server accepts connections on all container or host interfaces. For local health checks and browser opening, demo_instance.py prints and opens 127.0.0.1 instead because wildcard addresses are not usable browser destinations. In containers without host networking, expose the Dashboard port, for example -p 9202:9202.

To choose the Dashboard listen address:

python3 test/demo_instance.py \
  --ui \
  --ui-listen 0.0.0.0:9202

Explicit --ui opens the local default browser after the Dashboard is ready. In headless containers, remote shells, or CI, disable browser opening while still serving the Dashboard:

python3 test/demo_instance.py \
  --ui \
  --no-ui-open-browser

Use --no-ui to disable integrated Dashboard startup entirely, even if INSTANCE_UI_ENABLE=1 is set. The related environment variables are INSTANCE_UI_ENABLE, INSTANCE_UI_LISTEN, INSTANCE_UI_OPEN_BROWSER, and INSTANCE_UI_START_TIMEOUT_S; explicit CLI options take precedence over environment values.

Integrated mode connects the Dashboard to the same Resource Agent listen address and sample interval used by demo_instance.py. It always launches instance/resource_dashboard/dashboard_server.py with --no-auto-start, so demo_instance.py remains the only owner of the Resource Agent in the combined workflow. demo_instance.py also owns and cleans up only the Dashboard process it starts; if an already reachable Dashboard is reused, it is not terminated on Instance shutdown.

Dashboard startup, readiness, and browser-opening failures are warnings only. They do not stop the Instance; check the [demo_instance][ui] log lines for the generated command, readiness timeout or early-exit reason, and bounded stdout/stderr tails.