[yuecideng]

Description

Add optional owner-side ODS flow control and a bounded visual training/data-stream example. Consumers can wait for protected source credits, request refills and pace delivery while the host retains engine and optimizer ownership. The visual path starts training from published producer subbatches instead of waiting for a complete replacement generation.

Refs #788. Includes the flow-control foundation previously proposed in closed #823; targets main.

Core implementation

• ODS runtime: ODSFlowController provides bounded availability/quota waits, refill coordination and plan-based pacing. OnlineDataEngine.check_health() and get_flow_status() expose health and compact READY-state counters. Public immediate sampling and error behavior stays compatible.
• Sampling and lifecycle: remove the second copy after indexed TensorDict gathering and avoid redundant image padding. Preserve lock-held copying, independent returned storage and quota accounting. Serialize forkserver exit-status reads between owner and monitor to prevent false worker-exit errors.
• Existing BC integration: migrate the simple BC example to the shared controller, replacing its duplicated wait/refill/pacing loops. Keep the supporting collector and resource-sampler changes with their callers.
• Shared visual encoder: lazily export a trainable ResNet10/GroupNorm/spatial-embedding encoder adapted from pinned RLInf code, preserving attribution. The vision extra supplies matching torchvision; the example owns its BC head and loss.
• Runnable showcase: provide delivery diagnostics and single/dual-camera fixed-budget visual BC, real LeRobot/TorchCodec offline reads, bounded asynchronous recording, budget-aware prefetch, serial trace replay, source/anchor accounting, qualification, evaluation and report generators. Raw training/staging/serial ownership is bounded by C; recording queues and all accepted tail data are explicitly accounted for.

Review scope and evidence

Generated experiment outputs have been removed from this PR's final diff: 1,040 files, 174,638,781 logical bytes and 4,596,558 added text lines. Logs, plots, frozen source copies, machine-specific manifests and archives are retained at the immutable pre-cleanup commit, also backed up as codex/ods-stream-design-backup-20261024. Large datasets/checkpoints remain external local artifacts, as before.

The code review now contains 51 changed files and 16,932 additions, down from 1,089 files and 4,613,960 additions. The architecture design is consolidated from 722 to 195 lines. Generated output directories are ignored. EXPERIMENTS.md retains a compact evidence index, failed-attempt status and measured limitations.

Keep the controller, its simple BC callers, engine regressions, shared encoder and runnable example helpers together: they form active import/call dependencies. A later review split can place the ODS runtime/copy/lifecycle foundation below the visual encoder/showcase. The delivery entry points currently also own helpers used by visual training, so removing individual files requires extraction first.

The measurements support earlier usable data. They do not establish universal end-to-end acceleration, a diversity advantage or converged PickPlace learning. The formal 29/30 qualification result and its failed attempt remain unchanged; subsequent pilots and repairs are separate evidence. VLA/WAM integration is a follow-up.

Validation

• This cleanup changes artifact tracking and documentation only. Every retained executable/configuration/API-doc/test file is byte-identical to the backed-up implementation; all 1,040 locally preserved evidence files match their original Git blobs.
• Current py311 checks: Black 26.3.1 (black .), Python 3.10 parsing of retained changed modules, API coverage 2794/2794, context-map validation, 40 context tests, both training/delivery CLI help commands, local Markdown links and git diff --check pass.
• Existing implementation validation is linked in the immutable evidence index: focused production/model tests, reference encoder forward/gradient and image-dependent fitting checks, real SDK/video audits, physical pipeline smoke, and ten independently audited publication-pilot runs. These are historical runtime checks; no GPU experiment was rerun for this artifact-only cleanup.
• Test-necessity review: reuse existing production regressions; no new tests or generic example testing framework are added for removing generated files. Data-pipeline context was revised for the evidence location; other retained API/learning ownership guidance remains applicable.

Type of change

• New feature (non-breaking)
• Bug fix (non-breaking)

Checklist

• Run black . and proportional validation.
• Review affected documentation and project context.
• Keep public API documentation and production behavioral coverage.
• Preserve failed/negative experiment evidence outside the code diff.
• Document the optional vision dependency and reproduction commands.