[multi-lora] 2/n Tinker frontend: tinker==0.24.1 JSON/REST adapter

[yushengsu-thu]

Stacked on #2273; review this PR against tinker-compatible-backend.
#2365 adds the full-stack example and client-owned RL/SFT walkthrough.

Summary

Adds miles/ray/tinker_frontend, a protocol adapter for the exact
tinker==0.24.1 JSON core-loop surface. An unmodified official SDK client can
create LoRA training clients, submit explicit training operations,
checkpoint/resume, publish sampler weights, and sample through the existing
SGLang router.

The client still owns data construction, sampling policy, rewards,
advantages, and batch scheduling. This PR translates protocol requests;
#2273 owns operation ordering and Multi-LoRA execution.

Reviewer

official tinker==0.24.1 SDK
             │
             ▼
TinkerFrontendHTTPServer
  ├── /api/v1 routes, auth, and HTTP error mapping
  ├── wire.py: exact 0.24.1 JSON request/response models
  └── TinkerFrontend
      ├── state: sessions, models, futures, checkpoints, samplers
      ├── translation: Datum / Adam / results ↔ backend payloads
      │
      ├── TRAINING
      │     └── MultiLoraOperationController (#2273)
      │         └── MultiLoraOperationBackend
      │             └── ordered ledger
      │                 ├── forward[/backward]
      │                 │   └── MultiLoraOperationBatchFn → Megatron
      │                 └── optim/save/load/publish
      │                     └── Multi-LoRA control handlers
      │
      └── SAMPLING
            └── SamplingAdmission (weighted by num_samples)
                └── bounded SamplingTransport → SGLang router /generate

Sampling never enters MultiLoraOperationBatchFn or the legacy
dataset/reward rollout path.

Protocol and operation identity

• Health/readiness, capabilities, sessions, heartbeats, telemetry, model
  creation/info/unload, and asynchronous /retrieve_future polling.
• forward, forward_backward, per-call optim_step, immutable full-state
  save/load with optimizer, and latest-weight sampler publication.
• Base-model and ephemeral-LoRA /asample.
• Text-only shifted targets for cross_entropy, importance_sampling, and
  ppo; cross-entropy reports chunk-additive loss:sum,
  unmasked_tokens:sum, and loss_weight:sum.
• /api/v1/client/config keeps the 0.24.1 client on its JSON/httpx path.
  Other SDK versions are rejected at bootstrap; this does not implement the
  0.25+ protobuf protocol.

One SDK training client maps to one backend registration. Every training verb
forwards its SDK seq_id as that registration's operation ordinal.
Out-of-order chunks are gap-buffered by #2273. A frontend-rejected request is
recorded as terminal FAILED(user) so its already-spent ordinal cannot stall
later work.

Request IDs and fingerprints make identical retries replay-safe. Conflicting
content maps to 422, retryable pressure maps to 429 + Retry-After, and
expired futures map to 410. Terminal response bytes are retained before the
backend record is acknowledged; bounded fingerprint tombstones prevent an
expired identity from silently executing again.

Sampling safety and lifecycle

• Global admission is counted in sub-generations: one request consumes
  num_samples permits across all SDK clients.
• The router transport uses the same hard concurrency bound and matching
  connection pool, with no pool-acquisition timeout behind the admission gate.
• prompt + max_tokens is checked against an explicitly configured engine
  limit or one lazily discovered through the router and a healthy worker.
• Sampling parameters are rebuilt from an allowlist; client-supplied backend
  registration/routing identity never reaches SGLang.
• A multi-sample failure cancels and awaits sibling generations. Shutdown
  stops admission, drains sampling tasks, and then closes the transport.
• Session/future TTL maintenance cancels orphaned samples, resolves abandoned
  operation futures, and retires undelivered results into tombstones without
  freeing their execution identity.
• Ephemeral-LoRA sampling is version-checked before and after generation, so a
  stale or mid-flight republished sampler fails instead of knowingly returning
  cross-version output.

For non-loopback binds, an API key is mandatory. When configured, it protects
the SDK surface except health probes. Operator routes such as
/adapter_runs* and /info remain loopback-only even with the SDK key.

Supporting changes

• Adds backend facade projections needed by the frontend without exposing
  registry/ledger internals or the router URL.
• Adds terminal rejected-ordinal recording for failures after the SDK has
  already consumed a sequence number.
• Adds frontend launch, admission, context-limit, and retention flags.
• Pins tinker==0.24.1 in CPU CI so the real SDK contract tests run, and keeps
  pytest package arguments contiguous under pytest 9.1.

Validation

At current stacked head e87e4ef2d:

• H200 focused gate covering the frontend, operation backend, trainer/model
  initialization, arguments, predicates, and CI runner: 452 passed,
  3 skipped.
• Frontend CLI surface smoke passed: all seven current flags are present and
  the two removed legacy flags are absent.
• Pre-commit passed over all 40 effective PR files.
• Current GitHub pre-commit, CPU, 2-GPU, and all but one selected 4/8-GPU
  shards pass. The remaining 4-GPU H200 shard timed out after 1,800 seconds in
  test_inkling_small_4layer_lora_ci.py, which is outside this PR's effective
  diff; no Tinker frontend test failed.

Earlier official-SDK GPU acceptance covered Qwen pure SFT/RL and GPT-OSS
multi-adapter SFT/checkpoint and RL on 2×H200. The current head subsequently
adds sampling-lifecycle hardening; that code is covered by the focused H200
gate above, but the complete SDK GPU acceptance matrix was not rerun at this
exact head.

Scope and limits

• Requires [multi-lora] 1/n operation backend: explicit training with Tinker compatibility #2273. Multi-LoRA is the only wired training target; the unwired
  FullParameterExecutor seam is not exposed here and has no launch or
  GPU/E2E support claim.
• Text-only dense 1-D targets; no multimodal input, sparse CSR, nested top-K,
  or additional loss families.
• Full-state restore requires optimizer state. Named persistent sampler
  checkpoints and checkpoint TTL expiry are not implemented.
• Sampling serves latest weights only and admission is frontend-global, not
  per tenant. Request/output byte quotas and token-ID vocabulary upper-bound
  validation remain out of scope.
• Session, future, and tinker:// checkpoint catalogs are in-memory; process
  restart reconciliation for previously claimed operations is unchanged.

0821 liveness fixes merged forward from #2273: capped generate-failure tolerance in the driver, claimed-operation TTL backstop, FAILED child-runtime cooldown self-heal.

• 0821: SamplingClient.compute_logprobs() / sample(include_prompt_logprobs=True) now served natively (sglang logprob_start_len=0; topk_prompt_logprobs still rejected typed); README matrix updated; real-SDK contract tests + live GPU parity vs TrainingClient.forward (mean |Δ|=0.032, max 0.138).