v0.1.10-4: Add Unified KDN, LMCache, Provider, and Queue-Stage Observability

[rickisba]

Background

Later KDN Serving, Proxy queueing, and network-compute overlap work require reproducible stage-level measurements. The revised architecture distinguishes:

• KDN knowledge resolution and policy;
• KDN Remote Cache Serving;
• LMCache Compatibility Profile negotiation;
• Provider lookup/load/store execution;
• Instance-side LMCache cache loading and actual hits;
• Proxy queue and vLLM compute.

Tracing must preserve these boundaries instead of collapsing them into one cache-preparation duration.

Objective

Introduce a unified request trace and operation trace that records knowledge resolution, KDN serving, Provider/LMCache observations, queueing, and model execution without changing current routing, injection, or forwarding behavior.

Required Stages

When applicable, the schema should represent:

• KDN knowledge lookup and semantic resolution;
• Artifact compatibility evaluation;
• KDN endpoint selection;
• KDN Serving handshake/Profile negotiation;
• KDN Lookup queue and execution;
• Provider Lookup;
• KDN Store/Load/Retrieve/Prefetch task queue and execution;
• network transfer as observed by KDN, Provider, or LMCache;
• Instance-side LMCache load and confirmation;
• CachePlan/FusionPlan construction;
• Prefill queue and execution;
• Decode execution/completion;
• fusion preparation;
• fallback selection and execution.

Provenance Fields

Add explicit fields for:

• predicted_*;
• observed_*;
• actual_* or measured values;
• inferred values;
• source and source_endpoint;
• kdn_endpoint_id;
• provider_id and Provider generation;
• lmcache_profile_id;
• KDN serving protocol version;
• observation timestamp, freshness/age, and expiry;
• start, end, duration, queue duration, and execution duration;
• stage outcome;
• unsupported/incompatible capability;
• error and fallback reason.

Implementation Scope

• Define stable JSON-serializable Trace models.
• Keep request Trace and reusable serving-task Trace linkable through IDs.
• Distinguish KDN control decisions from KDN Serving hot-path execution.
• Distinguish KDN-reported results, Provider runtime observations, LMCache observations, Proxy inference, and vLLM measurements.
• Use monotonic clocks for durations and wall-clock timestamps only for external correlation.
• Preserve trace context across existing Proxy, KDN, and Instance boundaries.
• Allow disabled/sampled tracing without changing correctness.
• Keep Profile and Provider fields optional for text-only and Legacy paths.

Acceptance Criteria

• A request exports a deterministic, machine-readable stage timeline.
• A shared KDN serving task can be correlated with all waiting requests.
• Predicted, observed, measured, inferred, stale, unsupported, and skipped values cannot be confused.
• Every cache-state observation identifies source, Profile, generation, and freshness where available.
• KDN Serving queue time is separate from Provider operation time and Instance LMCache load time.
• Missing, unsupported, incompatible, stale, failed, and skipped stages are explicit.
• Stage durations are non-negative and use consistent units.
• Trace collection does not alter scheduling, injection, cache reuse, or forwarding outcomes.
• Text-only, Legacy KV, KDN remote KV, fallback, and unknown-future-Profile paths are representable.
• Unit tests cover serialization, ordering, shared task correlation, stale observations, unsupported capability, failed/skipped stages, and provenance separation.
• At least one CPU-only integration test produces an end-to-end timing breakdown using Mock KDN/LMCache Profiles.

Non-Goals

• Implementing the KDN Serving server.
• Implementing ExecutionGraph scheduling.
• Maintaining an authoritative Proxy or KDN physical block index.
• Claiming performance improvements in v0.1.10.
• Requiring an external tracing backend.

Parent Epic

• v0.1.10 Epic: KDN Contracts, LMCache Compatibility, and Observability Baseline #137

Dependencies

• v0.1.10-2: Define Storage-Neutral KDN, Cache Observation, Serving Task, and Queue State Models #139 for state/object naming.
• v0.1.10-3: Define Versioned KDN Server Contracts and LMCache Compatibility Profiles #140 for KDN protocol, Profile, Provider, and negotiation vocabulary.