Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ scripts/*
!scripts/demo/
scripts/demo/*
!scripts/demo/demo_markerless_calibration.py
!scripts/demo/demo_opencap_calibration.py
scripts/ralph_wiggum_viz/output/
tests/debug/*
*context.txt
Expand Down
30 changes: 21 additions & 9 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,29 @@ BSD-2-Clause licensed.

## What it solves

Multicamera 3D reconstruction requires knowing each camera's optical properties and its position in space.
Bundle adjustment finds these, but it needs a good starting point to converge.
Caliscope builds that starting point and refines it.
Triangulating a point from several cameras means knowing each camera's optical properties and where it sits in space.
Measuring that by hand is impractical, so it has to be recovered from video of the rig itself.

For each pair of cameras that both see the calibration target in the same frame, Caliscope estimates their relative position using PnP.
It chains these pairwise estimates transitively: if A-B and B-C are known, A-C is inferred.
The target never needs to be visible to all cameras at once.
The work runs in three stages.
Intrinsic calibration recovers focal length and lens distortion, one camera at a time.
Extrinsic calibration recovers where the cameras sit relative to each other, by bundle adjustment over the observations they share.
Anchoring then moves the solved rig into a frame you can use: vertical pointing up, distances in meters, floor at zero.

This approach supports flexible calibration targets.
A single ArUco marker on a sheet of paper can calibrate a wide capture volume.
For surround setups where cameras face inward from all directions, a charuco board printed on both sides of a rigid surface lets cameras on opposite sides link through shared points.
Everything is available two ways.
The desktop app gives visual feedback at each stage, and the [Scripting API](scripting.md) runs the same pipeline from Python.

## Calibrating with a target

Print a ChArUco board, an ArUco marker, or a chessboard, and record it moving through the volume.
The target never has to be visible to every camera at once, because Caliscope chains pairwise camera relationships transitively.
A single marker on a sheet of paper can calibrate a wide volume.
For surround rigs where cameras face inward from all sides, a board printed on both faces of a rigid surface links cameras that share no view.

## Calibrating without one

Cameras watching a moving person can be solved from the body alone, with no target.
Pose keypoints stand in for board corners, and the scene supplies the coordinate frame: an estimated vertical, one known distance for scale, and the floor.
This path needs intrinsics solved first, and it runs from the [Scripting API](scripting.md).

## Output

Expand Down
122 changes: 121 additions & 1 deletion docs/scripting.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,29 @@ For a two-sided board on a real substrate, pass the measured `thickness_cm` so b
charuco = Charuco.from_squares(columns=4, rows=5, square_size_cm=3.0, thickness_cm=0.6)
```

### Printing the board

The GUI's board panel also collects a board width, height, and unit.
Those dimensions never reach the calibration.
Both the GUI and `from_squares` pin the geometry to the measured square size, so the width and height set only the aspect ratio of the printed image.
A board built by `from_squares` and the same board configured in the GUI produce identical corner positions.

Use the full constructor when you want a printable board sized to your paper:

```python
charuco = Charuco(
columns=4, rows=5,
board_width=8.5, board_height=11.0, units="inch",
square_size_override_cm=5.4,
)
charuco.save_image("charuco.png")
charuco.save_mirror_image("charuco_mirror.png")
```

Print it, measure a square with calipers, and pass the measurement as `square_size_override_cm`.
Printers rescale.
The mirror image is the back face of a two-sided board.

## Step 2: Build a camera array from video metadata

```python
Expand Down Expand Up @@ -113,6 +136,17 @@ ext_points = extract_image_points_multicam(extrinsic_videos, tracker, timestamps
`frame_step` works on time-aligned moments, not raw frames.
`frame_step=10` processes every 10th synchronized moment.

If a camera recorded sideways, give it a quarter-turn count, the same value the GUI's rotate buttons set:

```python
ext_points = extract_image_points_multicam(extrinsic_videos, tracker, rotation_counts={2: 1, 3: -1})
```

Rotation decides only what the tracker sees.
Detected points come back in original image coordinates either way, so calibration results do not change.
Charuco, ArUco, and chessboard detection is already rotation-invariant and ignores the setting.
ONNX pose trackers need it because they were trained on upright people.

### Check camera pair coverage

Before calibrating, verify that camera pairs share enough observations:
Expand Down Expand Up @@ -185,6 +219,57 @@ volume = volume.align_to_object(sync_idx)
This applies a similarity transform (Umeyama algorithm) that aligns the world coordinate frame to the board's position at the chosen sync index.
Pick a frame where the board is at your desired origin.

### Rotate the axes

The board rarely lands with its axes where your lab convention wants them.
`rotate` turns the whole volume, cameras and points together.
It is what the GUI's X, Y, and Z buttons call:

```python
volume = volume.rotate("x", 90)
volume = volume.rotate("z", -90)
```

Positive angles follow the right-hand rule: counter-clockwise looking down the positive axis toward the origin.
The GUI buttons are fixed at 90 degrees, but any angle works here.

### Shift the origin

`translate` moves the whole volume by a fixed offset in meters:

```python
volume = volume.translate(z=0.02)
```

Shape and scale do not change, only where the rig sits in world coordinates.
Positive z raises everything.
This is the primitive `grounded` and `centered` are built on, exposed for the shifts they do not cover.

### Anchoring without a board

A markerless calibration has no board to align to, so the frame comes from the scene instead:

```python
volume = volume.oriented(up=vertical.up_per_cam) # +Z becomes vertical
volume = volume.scaled(CameraDistance(cam_a=0, cam_b=3, meters=4.2))
volume = volume.grounded() # floor at Z=0
volume = volume.centered() # XY origin at the camera centroid
```

Call them in that order.
`grounded` assumes Z is already vertical, and `centered` assumes the floor is already at zero.
`oriented` takes a per-camera up vector, which `estimate_vertical` produces from the video.

By default `grounded` rests the lowest triangulated point on Z=0, and that point is a marker, not the floor.
A toe marker rides above the ground on the shoe and its own thickness, so the floor ends up buried.
Measure that height and hand it over:

```python
volume = volume.grounded(lowest_point_height_m=0.02)
```

The marker then sits at 0.02 and the floor it stands on is zero.

## Step 7: Inspect results

```python
Expand All @@ -199,14 +284,28 @@ The report shows optimization status, reprojection error percentiles, per-camera
volume.save("capture_volume")
```

This writes three files to the directory: `camera_array.toml`, `image_points.csv`, and `world_points.csv`.
This writes `camera_array.toml`, `image_points.csv`, and `world_points.csv` to the directory, plus `constraints.toml` when the volume carries constraints.
To reload later:

```python
volume = CaptureVolume.load("capture_volume")
cameras = CameraArray.from_toml("capture_volume/camera_array.toml")
```

### Aniposelib export

Pose2Sim, anipose, and other aniposelib-compatible tools want their own TOML layout.
The GUI writes one to the workspace root every time it saves.
A script asks for it:

```python
volume.camera_array.to_aniposelib_toml("camera_array_aniposelib.toml")
```

Only posed cameras are written, with rotations as Rodrigues vectors.
`save` leaves this out on purpose.
Its directory stays a self-contained snapshot of the calibration, not a mix of snapshot and handoff files.

## Calibrating without intrinsics (experimental)

!!! warning "Experimental"
Expand Down Expand Up @@ -261,3 +360,24 @@ Detection is all-or-nothing, so a frame where any corner is cut off or covered c
A board with both counts even or both odd looks identical after a half turn, so the detector cannot tell the two orientations apart.
When two cameras see opposite orientations, corner ids reverse and triangulation silently pairs the wrong points.
A ChArUco or ArUco target does not have this problem.

## Matching the GUI

The two surfaces share their calibration code, so the same inputs give the same numbers.
The GUI just does more of the surrounding steps for you.

| GUI control | API equivalent |
|---|---|
| Board shape and square size | `Charuco.from_squares(columns, rows, square_size_cm)` |
| Board size and units | Print-only. Use the full `Charuco(...)` constructor. |
| Invert | `Charuco.from_squares(..., inverted=True)` |
| Save board PNG, mirror PNG | `charuco.save_image(path)`, `charuco.save_mirror_image(path)` |
| Camera rotate buttons | `rotation_counts={cam_id: turns}` at extraction |
| Calibrate | `calibrate_extrinsics(points, cameras, constraints)` |
| Refine intrinsics checkbox | `refine_intrinsics=True` |
| Filter outliers | `filter_percentile=2.5` |
| Set origin at frame | `volume.align_to_object(sync_index)` |
| X, Y, Z rotate buttons | `volume.rotate(axis, degrees)` |
| No GUI equivalent | `volume.translate(x, y, z)` |
| Saving a calibration | `volume.save(directory)` |
| `camera_array_aniposelib.toml` | `volume.camera_array.to_aniposelib_toml(path)` |
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "caliscope"
version = "0.11.3"
version = "0.11.4"
description = "Rapid and reliable multicamera calibration with GUI and scripting API"
authors = [{name = "Mac Prible", email = "prible@gmail.com"}]
license = { text = "BSD-2-Clause" }
Expand Down
1 change: 1 addition & 0 deletions scripts/demo/demo_markerless_calibration.py
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@
volume.camera_array,
world_points,
OUTPUT_DIR / "capture_volume_scene.py",
fps=60, # Pose2Sim Demo_SinglePerson footage
videos=videos,
wireframe=tracker_registry.wireframe_for(TRACKER_KEY),
)
Expand Down
Loading