SciGroveBeta
Robotics

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

Guanxiong Chen, Qianjun Xia, et al.

Featured August 4, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

This paper creates a smart AI system that watches real robots interact with objects, then automatically builds a perfect digital copy of that interaction in a computer game, making it much easier to teach robots new tricks.

In depth
The paper introduces Agentic Real2Sim, a novel framework that automates the conversion of real-world robot interaction recordings into high-fidelity, simulatable episodic twins. This is achieved by orchestrating a pipeline of specialized agents—visual processing, physical-prior inference, scene preparation, and simulator-in-the-loop optimization—all guided by a vision-language model (VLM). The framework significantly reduces the manual effort typically required to prepare real-world data for robotics tasks like policy learning and evaluation.

Key Takeaways

  • 1
    The framework automates the conversion of real-world robot interaction recordings into physically plausible digital twins for simulation, spanning rigid, deformable, and humanoid domains.
  • 2
    It leverages vision-language agents to make ambiguous decisions in the conversion process, orchestrating deterministic tools for visual processing, physical parameter inference, and scene setup.
  • 3
    The system supports interchangeable VLM backends, demonstrating that open-weight models can achieve comparable replay success rates to proprietary models at a significantly reduced cost, while also generalizing to various interaction types beyond rigid manipulation.

Conceptual Flow

HIGH LEVEL
1
Methodology: Automated Digital Twin Creation

The system uses smart AI agents to watch a real robot video, figure out all the details like shapes and physics, and then build a perfect digital copy for a computer game.

Real Robot Video
AI Agents Analyze
Digital Game Scene
2
Results: Scalable and Cost-Efficient Conversion

The new system successfully turns many real robot videos into digital copies, even with cheaper AI brains, and works for different kinds of robot actions.

Many Real Videos
Convert to Digital
Many Digital Copies
Works for Many Actions
Low Cost