Repository navigation
feat(vision): distortion, occlusion and synthetic frames - #24
Merged
Merged
Conversation
The vision package was geometry-only: an ideal pinhole, nothing but other pieces hiding a piece, plumb tags, no pixels. Pixel pipelines (a colour-blob VisionProcessor) could not run in the sim at all. - CameraLens: optional OpenCV five-coefficient distortion. project() and undistortPixel() (Newton), a fold-over guard, local magnification for the apparent radius. Without coefficients the arithmetic is the old pinhole, bit for bit. - CameraOccluders: field-frame and robot-frame boxes. SimCamera.observe with occluders casts 19 rays over each piece's disc against boxes and nearer pieces and reports getVisibleFraction(); a minimum fraction decides whether the piece is reported. SimTagCamera takes the same occluders. - AprilTagLayout.addTiltedTag: tags tilted up or down; tilt 0 is the plumb tag exactly. - SimFrameRenderer / SimFrame / FrameStyle: a seeded ray caster drawing the floor tiles, boxes, robot boxes, shaded spheres and tags as RGB bytes, with per-pixel ground-truth labels and a PNG writer whose tEXt chunk marks the frame as synthetic. No OpenCV dependency: the team's test wraps the bytes in a Mat. - FakeLimelight3A: distortion on the pixel outputs, robotBody(...) occlusion, seeded dropouts, and stdev_mt1/stdev_mt2 from the noise model. Docs say plainly that synthetic frames are drawings, not camera images, and that colour thresholds must never be tuned on them. Tests: 36 new (distortion, occlusion, tilted tags, renderer, Limelight fidelity); all 73 existing vision and Limelight tests pass unchanged.
A 640 x 400 frame of a real scene took about 90 ms, most of it in three places: - Sensor noise ran Box-Muller with StrictMath for every channel. It now reads a fixed 65,536-entry standard-normal table indexed by the same per-pixel hash, so it stays seeded and stateless. - A box with any corner behind the lens was left unbounded and tested on every ray, which is every robot box under a forward camera. Bounds are now cut at a near plane 0.001 in ahead of the lens, and a shape wholly behind it is never tested. - One thread. Rows of tiles now render on the common fork-join pool, each with its own copy of the per-ray state. Pixels are pure functions of the scene and write only their own slots, so the bytes do not depend on the thread count; a test renders on one thread and compares. Same scene: about 15 ms a frame. New test: a robot box reaching behind the lens is still drawn and still hides what is behind it.
Contributor
Author
|
Local checks (GitHub Actions cannot run while this repo is private; the org billing block). Run on this laptop, 2026-10-24, at
gitleaks 8.30.1 (the version the workflow pins, checksum verified). These are the same three checks the |
Contributor
Author
This was referenced Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Higher-fidelity vision in
org.horizon36596.simloop.vision: lens distortion, occlusion by field boxes, robot-body boxes and partly-covering pieces, tilted AprilTags, a seeded synthetic-frame renderer for running pixel pipelines, and Limelight fidelity options (distortion, robot-body occlusion, dropouts,stdev_mt1/mt2).Why
The camera was geometry-only: ideal pinhole, structures and the robot transparent, plumb tags, no pixels. A team whose robot runs a colour-blob
VisionProcessorcould not run that processor in the sim, and nothing could test undistortion or a structure hiding a ball.What is new (all additive; no public signature changed)
CameraLenswithDistortion(k1, k2, p1, p2, k3)(OpenCV order);project,undistortPixel,distortNormalized,localMagnification,getMaxValidRadius, coefficient gettersCameraOccluders(new)ObstacleBoxlistsSimCameraobserve(pieces, x, y, h, CameraOccluders): 19 rays per piece;withMinVisibleFraction;groundPointInRobotFrameundistortsCameraSightinggetVisibleFraction()(1.0 on the old path)AprilTagLayoutaddTiltedTag(...),Tag.getTiltUpRad(),Tag.getFaceNormal()SimTagCameraobserve/visibilitywithCameraOccluders; corners through the lens modelSimFrameRenderer,SimFrame,FrameStyle(new)FakeLimelight3A.Builderdistortion(...),robotBody(...),dropout(seed, p);stdev_mt1/mt2filled from the noise modelCost. Rows of 16 x 16 tiles render in parallel on the common fork-join pool, sensor noise comes from a seeded table, and image bounds are cut at the lens plane, so a 640 x 480 frame of a few dozen objects takes about 15 ms on a laptop. The bytes do not depend on the thread count: a test renders on a one-thread pool and compares.
Behaviour without the new options is bit-identical: every pre-existing vision and Limelight test passes unchanged.
A synthetic frame from
SimFrameRendererTest. Flat colours, one light. Not a camera image.Docs
docs/packages/vision.mdrewritten.limits.md,tested.md,features.md,index.md,llms.md,packages/index.md,packages/fakehardware.md,README.mdandsimloop-agents/AGENTS.mdupdated. They say that synthetic frames are drawings, not photographs, and that colour thresholds can be exercised on them but must never be tuned on them. Nothing published: the repo stays private and docs-publish was not run.How I tested it
./gradlew test :SimLoop:apiDocs: 1257 tests, 0 failures, 0 skipped; javadoc doclint clean./gradlew :SimLoop:docsSite: buildsbash tools/leak-gate/leak-gate.sh: passed (10 rules, 395 files)Consumer check: a team's real colour-blob
VisionProcessor(LAB threshold, watershed, ground projection, tracker) ran unchanged on these frames in its own headless tests, worst ball error 1.6 in across a lane runNew tests (37):
CameraLensDistortionTest9,SimCameraOcclusionTest8,SimFrameRendererTest10,SimTagCameraFidelityTest5,FakeLimelight3AFidelityTest5./gradlew testpasses on my machineNew or changed behaviour has a test
If I touched docs or public javadoc:
./gradlew :SimLoop:docsSitepassesChecklist