Skip to content

Real 2D-pose backend adapter interface (RTMPose / ViTPose) #33

Description

@bamdadd

Context

There's a rendering path (RendererBackend Protocol + PyrenderBackend, src/multicam_sim/dsl/render.py:100,110) and a depth path (DepthBackend, src/multicam_sim/dsl/depth.py:76), both open/closed Protocols. The pose layer emits ground-truth COCO-17 joints and, through the manifest, their per-camera 2D keypoints. What's missing is the loop that closes the evaluation: run a real 2D pose estimator on the rendered frames and compare its keypoints to ground truth.

What to do (larger)

Define a typed PoseEstimator Protocol (same open/closed style as RendererBackend): estimate(image) -> keypoints returning COCO-17 2D keypoints + confidences. Add an adapter for a real backend (RTMPose or ViTPose), gated behind an optional extra so the core stays import-light. Provide a helper that, for a scene, renders each camera/frame (via the existing RendererBackend), runs the estimator, and reports per-joint 2D error against the manifest's ground-truth uv (only where the joint is visible).

Acceptance criteria

  • PoseEstimator Protocol with a COCO-17 keypoint output type; the adapter implements it.
  • Heavy deps (mmpose/rtmpose or equivalent) are optional and guarded; tests skip cleanly when absent, and a fake/stub estimator exercises the compare-to-GT path on CPU.
  • The comparison helper returns per-joint pixel error keyed by camera and frame, restricted to visible joints.
  • ruff, mypy src, pytest -q green.

Pointers: dsl/render.py:100,107, dsl/depth.py:76, pose.py (COCO-17), manifest per_cam.uv. Renderer/pose design in DESIGN.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions