PhysiCalWorld In-Context Physics Calibration for World Models
On this page
TL;DR: We propose PhysiCalWorld, which calibrates world models to adhere to the dynamics of a novel test scene via prompting the model with an in-context interaction sequence collected within the test scene.
Abstract
Existing world models typically perceive a scene through current visual frame or a narrow history window. However, these observations may not reveal the physics properties that govern transition dynamics, such as an object’s mass distribution, joint damping, or surface friction. Without calibrating these physics factors for world models, predictions tend to regress toward an empirical prior and induce severe drift in novel test scenes.
To this end, we propose PhysiCalWorld, which calibrates action-conditioned world models to the instance-specific dynamics via in-context prompting, without fine-tuning or analytical physics modeling. Our method encodes an exploratory interaction clip collected within the test scene into a compact physics descriptor token, which grounds subsequent predictions in the scene’s actual physical behavior. PhysiCalWorld is trained end-to-end with the latent prediction loss alone, without physics labels. At inference time, the descriptor token is extracted once and can be reused across rollouts with negligible compute overhead.
We evaluate PhysiCalWorld across four environments covering diverse physics variations and show that it substantially improves rollout prediction accuracy and visual planning success rate. The learned descriptors also form a physically structured latent space and generalize beyond the physics ranges seen during training.
Method Overview
Model
Actions 
Under the Same Actions
(a) Uncalibrated World Model Imagining
Test Scene 
Dynamics
(b) Physics Calibration for World Models (Ours)
Benchmark Environments
We evaluate on four simulation environments with diverse physics variations, including mass 📦, mass distribution 🏗️, joint damping ⚙️, surface friction 🏷️, and air drag 💨. Identical actions can lead to very different future states.
Push-T under Varying Center of Mass. We vary the center of mass (CoM) of the T-shaped block, marked by the purple diamond in each video. The diamond is for visualization only and is hidden from the model; see “Start Obs” for the actual model input. Under identical pushing actions, different mass distributions lead to distinct rotational and translational motions.
Reacher under Varying Joint Damping. We vary the damping coefficients of a two-link robot arm. Under identical joint torques, different damping configurations produce distinct arm motions.
BeerPong under Varying Impact Energy Dissipation. We vary the table’s contact damping to control energy dissipation during impact. Under identical release actions, different damping values lead to distinct ball rebounds, with higher damping resulting in lower bounce heights.
FetchSlide under Multiple Physics Variations. We jointly vary puck mass, table surface friction, and aerodynamic drag. Under identical striking actions, these physical properties jointly determine the puck’s post-strike deceleration and sliding distance.
Experimental Results
Push-T
DINO-WM
DINO-WM
+ PhysiCal (Ours)
Ground Truth
All methods predict the same action sequence. Masks show the recorded ground-truth object locations at each prediction frame after the initial frame. Toggle Show context to view the shared exploratory interaction.
+ PhysiCal (Ours)
Each row is one method: its world-model rollout (WM Rollout) and the executed trajectory (Exec Traj) for the same planning example. Masks mark the fixed goal object locations. DINO-WM-based and LeWM-based methods use separate example selections. Toggle Show context to view the shared exploratory interaction.
Push-T Cosine Similarity
Physical structure emerges without physics labels. Select a pair to compare its descriptors.
Each cell averages similarities between interaction clips. Diagonal cells compare distinct clips at the same CoM.
Push-T Inference Speed
A reusable physics descriptor keeps inference close to the base world model.
Push-T · Mean latency (ms) · Lower is better
One RTX PRO 6000 Blackwell GPU · FP32, TF32 disabled. Mean of 100 timed calls after 20 warm-ups across 10 episodes. Includes initial observation and context encoding; excludes decoding and data I/O.
Reacher
DINO-WM
DINO-WM
+ PhysiCal (Ours)
Ground Truth
All methods predict the same action sequence. Masks show the recorded ground-truth object locations at each prediction frame after the initial frame. Toggle Show context to view the shared exploratory interaction.
+ PhysiCal (Ours)
Each row is one method: its world-model rollout (WM Rollout) and the executed trajectory (Exec Traj) for the same planning example. Masks mark the fixed goal object locations. DINO-WM-based and LeWM-based methods use separate example selections. Toggle Show context to view the shared exploratory interaction.
BeerPong
DINO-WM
DINO-WM
+ PhysiCal (Ours)
Ground Truth
All methods predict the same action sequence. Masks show the recorded ground-truth object locations at each prediction frame after the initial frame. Toggle Show context to view the shared exploratory interaction.
LeWM uses the TC-LeWM variant in BeerPong, as in the paper.
+ PhysiCal (Ours)
Each row is one method: its world-model rollout (WM Rollout) and the executed trajectory (Exec Traj) for the same planning example. Masks mark the fixed goal object locations. DINO-WM-based and LeWM-based methods use separate example selections. Toggle Show context to view the shared exploratory interaction.
LeWM uses the TC-LeWM variant in BeerPong, as in the paper.
FetchSlide
DINO-WM
DINO-WM
+ PhysiCal (Ours)
Ground Truth
All methods predict the same action sequence. Masks show the recorded ground-truth object locations at each prediction frame after the initial frame. Toggle Show context to view the shared exploratory interaction.
+ PhysiCal (Ours)
Each row is one method: its world-model rollout (WM Rollout) and the executed trajectory (Exec Traj) for the same planning example. Masks mark the fixed goal object locations. DINO-WM-based and LeWM-based methods use separate example selections. Toggle Show context to view the shared exploratory interaction.
Extrapolation
We investigate whether PhysiCalWorld can extrapolate to environments where physics parameters lie outside the training distribution.
In Reacher, we sweep the first joint’s (should joint) damping coefficient beyond its training range while holding the second joint’s damping fixed.
Mean ± SEM across 75 episodes per bin (15 damping values × 5 episodes).Open-Loop rollout. IoU is computed over the foreground mask of the robotic arm, averaged over episodes. Joint 2's damping is fixed.