Plumb: An Agent Harness for Inverse Physics of Dynamic Scenes

Motivation

Can a coding agent reconstruct a physical event from video as a simulation? This requires choosing objects, geometry, physical parameters and an initial state whose simulated motion agrees with the observations. We gave Codex a MuJoCo environment and asked it to reconstruct synthetic and real-world videos. Inspecting the resulting scenes and fitting scripts revealed errors in object counts, camera motion and the 3D layout of the scene.

01Object counts and geometry. In this real-world marble run, one ramp contains 14 dominoes. Codex reconstructed 21; Plumb reconstructed 14. The discrepancy changes both the scene geometry and the possible sequence of contacts.

02Camera motion. In this billiards video, the handheld camera moves slightly near the end of the clip. Codex does not capture such small camera motion and renders the whole video from one fixed camera. Plumb estimates a camera pose for each frame and follows the motion.

03Perceiving the scene in 3D. Codex has no 3D perception of the scene: it only adjusts its reconstruction until the rendered RGB image matches the input view. In this double pendulum, Codex's scene lines up with the input view, but its field of view is far too narrow, its scale is off, and the lanterns and the room sit at the wrong depths. From an unseen view to the side, the reconstruction falls apart. Plumb takes its camera and depth from Pi3X, so its scene stays much closer to the ground truth from the side as well.

Method

In Plumb, the coding agent receives the video, a MuJoCo sandbox and the task requirements, and writes everything that defines the reconstruction: the MJCF scene, its initial state, the physical parameters and the fitting code. We find that frontier models do not need a complex harness; an overly complex harness can even hurt.

• Foundation models. SAM3 segments and tracks objects from text prompts or example boxes. Pi3X estimates per-frame depth, camera intrinsics and camera poses at an approximately metric scale.
• Measurements. From these outputs, helper commands derive a camera on a fitted support plane, the slope of ramps and tables, the size and 3D path of each tracked object, the motion of a moving camera, and 2D object tracks.
• Simulation checks. Any scene can be rolled out on the video's timeline, rendered through its camera and compared with the tracks, in image space and in depth.
• Delivery. A run-script template renders the preview video, and a format check confirms that the scene runs passively over the full video.

Results

We evaluate on 50 videos, 25 from PhysInOne and 25 from Physion v1.5, selected by category. We removed videos whose ground truth conflicts with physical common sense and videos that lack the annotations needed for evaluation. We adjusted the videos' playback rates to match their simulation timing.

All agent calls in this comparison use GPT-6.1 Sol at high reasoning effort.

Comparison with Codex

We report mask intersection over union (IoU) and the average displacement error (ADE) of the moving objects, in 2D (pixels) and in 3D (metres), against dataset annotations. The second-view metrics render each reconstruction from an unseen view, a camera the agent never saw. Physics accuracy is the fraction of 279 physical quantities, such as launch speed, restitution and swing period, that a reconstruction recovers within a factor of 1.25 of the ground truth.

What's next

• Use more foundation models. Plumb currently uses only the camera and depth from Pi3X. Other foundation models, such as FoundationPose for object pose, could also be added.
• Predict physical parameters better. Physical parameters are not yet predicted well: even with Plumb, only about a third of the physical quantities, such as restitution, deceleration and mass ratios, fall within a factor of 1.25 of the ground truth.
• Extend the scene models. The current system models rigid-body dynamics. Deformable objects, cloth and fluids would require additional representations, simulation tools and evaluation criteria.

Citation

To cite this work, please use:

@misc{liang2026plumb,
  title        = {Plumb: An Agent Harness for Inverse Physics of Dynamic Scenes},
  author       = {Liang, Xiaoqian and Chen, Hongyuan and Huang, Binbin and Chen, Anpei},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://yubel426.github.io/Plumb/}
}