feat(grid): hot reload route overlays

[nerdalert]

Summary

Adds draft/WIP hot-reload support for the grid_route filter.

This introduces an overlay-file mode where grid_route reads a Grid-generated grid-config.json file, builds an in-memory route snapshot, and can update that snapshot in-process when the file changes. The intent is to let Grid update provider candidates without requiring a Praxis AI pod restart, rollout restart, or full proxy pipeline reload.

This PR is intentionally stacked on top of #339 and should remain draft until grid_route lands.

Why this exists

Grid owns provider discovery, provider health/freshness, and route ordering. Praxis AI owns request-time routing.

The handoff between them is the Grid routing overlay:

Grid watches CRDs + provider state
→ Grid renders grid-config.json
→ Kubernetes projects the ConfigMap into the Praxis AI pod
→ grid_route watches the overlay file
→ grid_route validates the new overlay
→ grid_route swaps its in-memory route snapshot
→ new requests use the new routes
→ old in-flight requests finish on the old snapshot

This PR implements the Praxis AI side of that handoff.

High-level behavior

grid_route now supports two modes:

1. Static mode

   Existing behavior. Candidates are declared inline in the Praxis config and remain fixed for the lifetime of the filter.

2. Overlay mode

   Candidates are loaded from overlay_file, normally /etc/grid/grid-config.json. The filter can watch the overlay file parent directory and reload candidate state when the file changes.

The two modes are mutually exclusive.

Reload algorithm

At startup:

read overlay file with a bounded reader

  → parse JSON
  → validate localSite + candidates
  → build RouteSnapshot
  → store RouteSnapshot in ArcSwap
  → if reload is enabled, start watcher thread

On filesystem event:

  watch parent directory, not the file path directly
  → accept relevant create/modify/remove/rename-style events
  → debounce events
  → read overlay file with MAX_OVERLAY_SIZE + 1 bound
  → reject oversized/unreadable files
  → hash file contents
  → if hash unchanged, do nothing
  → parse and validate overlay
  → if valid, atomically swap ArcSwap<RouteSnapshot>
  → if invalid, retain last-known-good snapshot

On request:

  load current ArcSwap snapshot
  → extract requested model/tool/capability
  → choose candidate from the loaded snapshot
  → write route metadata
  → set downstream load_balancer cluster

The request path does not read files or talk to Kubernetes.

Kubernetes ConfigMap behavior

The watcher intentionally watches the overlay file's parent directory. This is needed because Kubernetes projected ConfigMap updates are usually delivered as symlink swaps involving ..data, timestamped directories, rename events, or create/remove events rather than a simple direct file modification.

The implementation avoids canonicalizing the overlay path so projected-volume symlink behavior is preserved.

Important deployment requirement:

• mount the ConfigMap as a directory;
• do not use subPath;
• subPath bypasses the projected-volume symlink swap mechanism and will not provide reliable hot-reload events.

Safety properties

This PR is designed around last-known-good behavior:

• startup fails if the initial overlay cannot be read or validated;
• after startup, invalid reloads do not replace the active snapshot;
• unreadable/missing files after startup retain the previous snapshot;
• oversized files are rejected through bounded reads;
• same-content rewrites do not swap snapshots;
• in-flight requests can continue using the old Arc while new requests observe the new snapshot;
• credential token values are rejected from overlays; only credential references are allowed.

What hot reload can change

The overlay can change:

• provider candidates;
• candidate freshness;
• candidate order;
• candidate cluster choice;
• credential reference metadata.

The overlay cannot change:

• downstream load_balancer cluster definitions;
• cluster endpoints;
• TLS/SNI transport configuration;
• filter chain shape;
• credential secret values.

Every cluster name that may appear in any overlay version must already exist in the downstream load_balancer config. If an overlay references an unknown cluster, it is syntactically valid and may become active, but requests selecting that cluster will fail downstream.

Files changed

• filters/src/grid/overlay.rs

  • overlay wire types;
  • RouteSnapshot;
  • bounded reads;
  • hash comparison;
  • watcher lifecycle;
  • last-known-good reload behavior.

• filters/src/grid/route.rs

  • overlay mode config;
  • reload config;
  • ArcSwap-backed snapshot selection;
  • static/overlay validation matrix.

• examples/configs/grid-route-overlay.yaml

  • example overlay-backed grid_route config.

• tests/schema/tests/suite/examples/ai/grid_route.rs

  • schema-level overlay routing test;
  • schema-level in-process hot-reload test.

Validation

Ran locally:

  cargo test -p praxis-ai-filters grid
  cargo test -p praxis-tests-schema grid_route
  cargo check -p praxis-ai-proxy
  cargo +nightly-2026-03-28 fmt --all -- --check
  git diff --check

Draft review focus

This is intentionally still draft/WIP.

1. watcher lifecycle and shutdown guarantees;
2. startup read/watch race behavior;
3. Kubernetes ..data symlink-swap handling;
4. bounded file reads;
5. last-known-good behavior before ArcSwap::store;
6. request-path snapshot guard lifetime;
7. no token values accepted in overlays or written to metadata;
8. overlay/static validation matrix;
9. same-content rewrite no-op behavior;
10. full Grid/Praxis E2E proof before merge.

Known remaining work

Before this is ready to merge:

• run the full Grid GLB/hot-reload E2E;
• prove reload without Praxis pod/container restart;
• prove backend attribution before and after overlay change;
• prove invalid overlay retains last-known-good routing;
• confirm watcher thread/resource cleanup under repeated rebuilds;
• update docs after the implementation shape is finalized.