Repository navigation
ingest: vendor buckets → conversion jobs → staff review → lance_commit (store, watcher, AWS deploy) - #720
Draft
SaveAanchal wants to merge 3 commits into
Draft
ingest: vendor buckets → conversion jobs → staff review → lance_commit (store, watcher, AWS deploy)#720SaveAanchal wants to merge 3 commits into
SaveAanchal wants to merge 3 commits into
Conversation
SaveAanchal
force-pushed
the
aanchal/ingest
branch
from
October 21, 2026 06:27
21a96ac to
1816d7c
Compare
SaveAanchal
force-pushed
the
aanchal/mcap-to-lance
branch
from
October 21, 2026 06:27
eaa8bb1 to
47ebba9
Compare
SaveAanchal
changed the title
RyanPCo
force-pushed
the
aanchal/ingest
branch
from
October 22, 2026 02:04
5820f84 to
d6f65b6
Compare
RyanPCo
changed the title
The pipeline half of #720 as Aanchal wrote it, squashed: the bucket store (R2 and local, size-bound uploads, completion snapshots, staged results by digest), staging with a reviewer diff, format detection (MCAP first), the watcher pass and its SkyNet job, plus the mcap_to_lance --mano-model-dir option and the ffmpeg mute in validate.py. The dashboard half moves to its own branch. This commit is her code as of origin/aanchal/ingest 5820f84; the next commits adapt it to the staged-delivery committer and AWS, so it does not pass tests on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
The pipeline now feeds lance_commit instead of writing Lance itself: - store: logical keys route to buckets. deliveries/<vendor>/ lives in that vendor's ingest-<vendor> bucket (a write key for exactly that bucket lets a vendor upload with its own S3 tools, finishing with _done.json); pipeline/ lives in the private ingest bucket, with vendors.json listing the vendor buckets. copy_out copies an object out server-side (R2 managed copy), so hand-offs never leave R2. - stage: prepare() writes a staged delivery (commit.stage_delivery) plus stage.json (fingerprint, converter, settings, sha256 and size of every file) and the reviewer's diff.json. commit_staged, the base-state pins and the cut-off-commit recovery go: the committer owns commits. - convert_job: one delivery's conversion as a job (download, prepare, store the stage, record the outcome under pipeline/jobs/), launched by launch.py on AWS Batch or in-process for tests and local runs. - watch: a pass collects job outcomes, submits new or re-uploaded deliveries (at most --max-in-flight at once; while a vendor has nothing approved at the current converter revision, only --canary of its deliveries), and on approval copies the raw files to raw_v2/<vendor>/<delivery>/<stage>/ and the stage to lance_staging/<lab>/<vendor recording id>/ with provenance in meta.json, then deletes the delivery and stage. It follows handed-off deliveries to committed or commit_rejected. - run: the box's loop, a pass and then the committer every 5 minutes. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
Replaces the SkyNet job: - aws/Dockerfile: one CUDA image for the Batch conversion jobs and the box's loop (static ffmpeg with SVT-AV1, the locked venv, MANO from its pinned commit; the license-gated model files come in as a build secret). aws/setup_batch.py creates a Spot compute environment of 1-GPU g6/g5.2xlarge instances (min 0, max 448 vCPU = 56 jobs), its queue and the job definition, with Spot reclaims retried; --dry-run prints the calls. aws/README.md is the runbook, with measured cost: ~4 GPU machine-hours and 2.3 GB of output per footage-hour. The lab account's G-instance quotas are 0 in us-east-2 and need raising first. - onboard.py: an admin creates ingest-<vendor>, its CORS and abandoned- upload lifecycle rules, an account token limited to that bucket (S3 keys printed once) and the vendors.json entry, from their own machine. - review.py: list, show, approve / decline (one, several or every ready delivery of a vendor), each decision bound to the status's stage. The R2 integration test runs against the ingest buckets. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
RyanPCo
force-pushed
the
aanchal/ingest
branch
from
October 22, 2026 14:06
d6f65b6 to
df8c054
Compare
RyanPCo
pushed a commit
that referenced
this pull request
Oct 22, 2026
The dashboard half of #720 as Aanchal wrote it, squashed: accounts (scrypt hashes, created by staff), login with CSRF, throttling and strict cookies, vendor and staff views, presigned multipart uploads bound to their declared sizes, staff approve / decline one at a time or in bulk, and the standard-library upload CLI it serves. Adds the dashboard extra (flask) to pyproject.toml. Her code as of origin/aanchal/ingest 5820f84; the next commit points it at the per-vendor buckets and the new watcher states. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
RyanPCo
force-pushed
the
aanchal/mcap-to-lance
branch
from
October 22, 2026 14:06
4815a4f to
dca8ff9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #719. The pipeline half of the original #720; the dashboard moved to its own PR on top.
Aanchal's pipeline code is the first commit (authored by her, as of 5820f84); the two commits on top adapt it to the decisions in the ingest plan.
What changed from the original
deliveries/<vendor>/…lives in that vendor's own bucketingest-<vendor>(a write key for exactly that bucket lets a vendor upload with its own S3 tools, finishing with_done.json);pipeline/…lives in the privateingestbucket, withvendors.jsonlisting vendor buckets. Therldbread token is public, so nothing private lives there.stage.prepare()writes a staged delivery (commit.stage_delivery) plusstage.json(fingerprint, converter, settings, sha256 + size per file) anddiff.json.commit_staged, the base-state pins and cut-off-commit recovery are gone. On approval the watcher copies the raw files torldb/raw_v2/<vendor>/<delivery>/<stage>/and the stage torldb/lance_staging/<lab>/<vid>/(meta.json last, with provenance), both server-side, andlance_commitcommits it. A recording already in Lance needslance_commit --replace(statuscommit_rejecteduntil then).convert_job.pyconverts one delivery and records its outcome underpipeline/jobs/;launch.pyruns it on AWS Batch (or in-process for tests). The watcher caps jobs in flight (56) and, until a vendor has an approval at the current converter revision, converts only a canary of 3 of its deliveries.run.pyis the box's loop (a pass, then the committer, every 5 min).aws/Dockerfile,aws/setup_batch.py(Spot GPU compute environment, queue, job definition) andaws/README.md(runbook) replaceskynet/.onboard.py(admin, from their own machine: bucket, CORS, abandoned-upload lifecycle, bucket-scoped token, vendors.json entry) andreview.py(list / show / approve / decline, bulk) .Measured
One 3-minute Aria recording: 735 s on an A40 + 8 cores (MANO 553 s), about 4 machine-hours per footage-hour and 2.3 GB output per footage-hour. MANO on CPU alone is ~14x realtime, so jobs take a GPU.
Not run yet
No AWS or Cloudflare call has run for real (Batch submit, onboarding,
tests/integration/test_ingest_r2.py): no write credentials here. The lab AWS account's G-instance quota is 0 in us-east-2; the Batch/Spot rollout is deferred for now.Tests
tests/unit/test_ingest_{store,stage,formats,watch,tools}.py: 137 pass, including the full loop with in-process jobs and the committer.https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd