Repository navigation
ingest: restack of #720 onto the #719 replacement - #744
Draft
SaveAanchal wants to merge 7 commits into
Draft
SaveAanchal wants to merge 7 commits into
SaveAanchal wants to merge 7 commits into
Conversation
The pipeline half of #720 as Aanchal wrote it, squashed: the bucket store (R2 and local, size-bound uploads, completion snapshots, staged results by digest), staging with a reviewer diff, format detection (MCAP first), the watcher pass and its SkyNet job, plus the mcap_to_lance --mano-model-dir option and the ffmpeg mute in validate.py. The dashboard half moves to its own branch. This commit is her code as of origin/aanchal/ingest 5820f84; the next commits adapt it to the staged-delivery committer and AWS, so it does not pass tests on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
The pipeline now feeds lance_commit instead of writing Lance itself: - store: logical keys route to buckets. deliveries/<vendor>/ lives in that vendor's ingest-<vendor> bucket (a write key for exactly that bucket lets a vendor upload with its own S3 tools, finishing with _done.json); pipeline/ lives in the private ingest bucket, with vendors.json listing the vendor buckets. copy_out copies an object out server-side (R2 managed copy), so hand-offs never leave R2. - stage: prepare() writes a staged delivery (commit.stage_delivery) plus stage.json (fingerprint, converter, settings, sha256 and size of every file) and the reviewer's diff.json. commit_staged, the base-state pins and the cut-off-commit recovery go: the committer owns commits. - convert_job: one delivery's conversion as a job (download, prepare, store the stage, record the outcome under pipeline/jobs/), launched by launch.py on AWS Batch or in-process for tests and local runs. - watch: a pass collects job outcomes, submits new or re-uploaded deliveries (at most --max-in-flight at once; while a vendor has nothing approved at the current converter revision, only --canary of its deliveries), and on approval copies the raw files to raw_v2/<vendor>/<delivery>/<stage>/ and the stage to lance_staging/<lab>/<vendor recording id>/ with provenance in meta.json, then deletes the delivery and stage. It follows handed-off deliveries to committed or commit_rejected. - run: the box's loop, a pass and then the committer every 5 minutes. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
Replaces the SkyNet job: - aws/Dockerfile: one CUDA image for the Batch conversion jobs and the box's loop (static ffmpeg with SVT-AV1, the locked venv, MANO from its pinned commit; the license-gated model files come in as a build secret). aws/setup_batch.py creates a Spot compute environment of 1-GPU g6/g5.2xlarge instances (min 0, max 448 vCPU = 56 jobs), its queue and the job definition, with Spot reclaims retried; --dry-run prints the calls. aws/README.md is the runbook, with measured cost: ~4 GPU machine-hours and 2.3 GB of output per footage-hour. The lab account's G-instance quotas are 0 in us-east-2 and need raising first. - onboard.py: an admin creates ingest-<vendor>, its CORS and abandoned- upload lifecycle rules, an account token limited to that bucket (S3 keys printed once) and the vendors.json entry, from their own machine. - review.py: list, show, approve / decline (one, several or every ready delivery of a vendor), each decision bound to the status's stage. The R2 integration test runs against the ingest buckets. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
Fits the copied #720 onto the restacked converter and current committer: - the pin records `source` (mcap) and the hand-off stages under <lance_staging>/<source>/<vendor recording id>/; the loop commits per source; - diff reads the recording's own steps table (lab, embodiment, end_effector); - labels that name no annotation_source belong to the episode's lab, so a re-export withdraws the lab's labels and leaves other sources' alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The watcher copied an approved stage to <lance_staging>/mcap/<vendor recording id>/, where two vendors (or labs) with the same vendor id would overwrite each other. Hand off under the pin's recording_id, the folder the committer now expects for self-describing deliveries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follows the mcap recording-id change: the conversion job passes the delivery's Ref.vendor through prepare_delivery to stage.prepare and the converter, so the recording id is mcap/<vendor>/<vendor_recording_id> for the bucket the files came through, whatever lab the metadata names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stage.prepare validates with allow_missing, so a delivery that lists scene or objects in missing_metadata is converted and waits for review instead of being rejected; the validator's waiver warnings are kept as report["waived"] (also in the pin) for staff to weigh when approving. task_name, operator and lab still cannot be waived and are rejected as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces #720. #720's three commits restacked onto
aanchal/mcap-lance-v2(the #719 replacement), plus what the new layout needs:<staging>/mcap/<recording_id>/, and the loop commits per source.prepare, so the id isrecording_id("mcap/<vendor>", vid)and can't be steered by the file's own metadata (depends on the contract change in the parent PR).allow_missing, so a waivedscene/objectsis staged and listed inreport["waived"]instead of rejected.task_name,operatorandlabstill can't be waived.validate.py(muting ffmpeg logs) was applied by hand, since main reorganised that file.Test plan
pytest tests/unit(all pass excepttest_infra_port_wiring.py::test_pi_loss_is_reduced_over_the_packed_width, which needsopenpi)🤖 Generated with Claude Code