OpenJaws: An Open Hardware UMI Framework
with Stereo Localization for Generalist Manipulation

Localization builds on a stereo camera and classical triangulation, which determines the 3D position of a scene point by intersecting the rays projected through two calibrated camera centers. Triangulating corresponding pixels across two views separated by a known physical baseline resolves absolute scale — so, unlike a monocular camera, the rig recovers metric trajectories from vision alone, with no IMU and no external infrastructure.

Handheld stereo data collection with OpenJaws. The purple curve is the recovered end-effector trajectory over the full recording. The insets are the rectified left (blue) and right (orange) frames captured at the starting pose, with SIFT correspondences drawn between them, and rays running from each lens to its image. The fiducial cube is used only for benchmarking, not for policy data collection; the point cloud, from FoundationStereo, is for visualization only.

SLAM benchmarks exist for autonomous driving and indoor scene reconstruction, but to the best of our knowledge UMI-Loc-Bench-v1 is the first SLAM localization benchmark in the context of UMI or robotic manipulation for policy learning. It contains 90 handheld demonstrations — 10 for each of nine tasks — spanning short-horizon pick-and-place, constrained retrieval, contact-rich insertion, and non-prehensile manipulation, across multiple scenes. Ground truth comes from a static ArUco cube that is masked out of both camera streams before any method runs, and each method is scored on translational and rotational ATE after a scale-fixed Umeyama SE(3) alignment, plus wall-clock runtime.

Example rollouts from UMI-Loc-Bench-v1, with the recovered (green and black) end-effector trajectories for each task.

We evaluate representative SLAM and SfM backends spanning stereo, monocular, and monocular-inertial settings: COLMAP as a stereo-rig reconstruction, cuVSLAM (CUDA-accelerated SLAM meant for realtime applications), ORB-SLAM3 (the feature-based backend adopted by UMI), the learned monocular video-pose model ViPE, and UMI-SLAM — UMI's modified ORB-SLAM3 mono+IMU pipeline, which loads a prerecorded map and uses a known-size fiducial marker to stabilize initialization. Monocular baselines all use the left stereo image.

Coverage and Runtime

Stereo methods track essentially the full trajectory, whereas ORB-SLAM3 mono+IMU and UMI-SLAM average only 9% and 37% coverage. COLMAP is by far the slowest at 836 s per run against cuVSLAM's 34.8 s, so we use cuVSLAM for the policy-learning experiments.

Accuracy

COLMAP reaches 2.9 mm / 0.4° against 18.4 mm / 4.0° for UMI-SLAM, and among the pure-stereo backends it is the most accurate over full trajectories (5.2 mm / 0.8°, versus 7.4 mm for ORB-SLAM3 and 12.4 mm for cuVSLAM). The learned monocular model ViPE recovers geometrically faithful trajectories (3.5 mm / 0.7° with a fitted scale) but not metric ones — 48.2 mm at native scale. This is because a monocular camera cannot resolve absolute scale and the UMI-style wrist camera data we evaluated on is out-of-distribution for the room-scale data used for training ViPE.

We train Diffusion Policy with a relative Cartesian end-effector action space on 200 demonstrations for block stacking and 50 for pick-and-place, localized with stereo-cuVSLAM. The gripper attaches directly onto the robot arm, requiring no recalibration between collection and deployment. Both policies reach a 90% success rate over 10 trials under varied initial conditions.

OpenJaws turns UMI demonstrations into simulation digital twins agentically: the GPT-6 (Astra) agent gets only the demonstration and sim-ready assets — not the objects' ground-truth poses — and recovers the scene graph itself. On the gear-insertion task it produces working twins for 7 of 10 demonstrations, automatically recovering the signed-distance-field collision geometry this contact-rich task needs.