Robotics

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

Guanxiong Chen, Qianjun Xia, et al.

Featured August 4, 2026

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

This paper creates a smart AI system that watches real robots interact with objects, then automatically builds a perfect digital copy of that interaction in a computer game, making it much easier to teach robots new tricks.

In depth

The paper introduces Agentic Real2Sim, a novel framework that automates the conversion of real-world robot interaction recordings into high-fidelity, simulatable episodic twins. This is achieved by orchestrating a pipeline of specialized agents—visual processing, physical-prior inference, scene preparation, and simulator-in-the-loop optimization—all guided by a vision-language model (VLM). The framework significantly reduces the manual effort typically required to prepare real-world data for robotics tasks like policy learning and evaluation.

Key Takeaways

1
The framework automates the conversion of real-world robot interaction recordings into physically plausible digital twins for simulation, spanning rigid, deformable, and humanoid domains.
2
It leverages vision-language agents to make ambiguous decisions in the conversion process, orchestrating deterministic tools for visual processing, physical parameter inference, and scene setup.
3
The system supports interchangeable VLM backends, demonstrating that open-weight models can achieve comparable replay success rates to proprietary models at a significantly reduced cost, while also generalizing to various interaction types beyond rigid manipulation.

Conceptual Flow

HIGH LEVEL

Methodology: Automated Digital Twin Creation

The system uses smart AI agents to watch a real robot video, figure out all the details like shapes and physics, and then build a perfect digital copy for a computer game.

Real Robot Video

AI Agents Analyze

Digital Game Scene

Results: Scalable and Cost-Efficient Conversion

The new system successfully turns many real robot videos into digital copies, even with cheaper AI brains, and works for different kinds of robot actions.

Many Real Videos

Convert to Digital

Many Digital Copies

Works for Many Actions

Low Cost

This breakdown was generated by SciGrove. Get the same analysis — intuition, storyboard, peer review, a runnable prototype and a glossary — on any paper you upload or paste a DOI for.