Robotics

VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Yen-Jen Wang, Jiaman Li, Sirui Chen, Takara E. Truong, Pei Xu, Pieter Abbeel, Rocky Duan, Koushil Sreenath, Angjoo Kanazawa, Carmelo Sferrazza, Guanya Shi, Karen Liu

Featured July 1, 2026

Original Paper

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

Robots learn to move and pick things up by practicing in realistic computer-generated rooms, using a special camera view and simple instructions, then they can do it in the real world.

In depth

The paper introduces a novel pipeline for training humanoid robots to perform complex loco-manipulation tasks by generating vast amounts of synthetic vision-language-kinematics (VLK) data. This data is created by synthesizing robot interactions within photorealistic 3D environments reconstructed using 3D Gaussian Splatting, effectively bridging the gap between simulation and real-world deployment for perception-based control.

Key Takeaways

1
The authors developed an automated pipeline that generates 48,000 paired VLK trajectories in reconstructed 3D scenes, overcoming the data scarcity for humanoid loco-manipulation.
2
3D Gaussian Splatting is leveraged to create photorealistic, metric-scale indoor environments, providing a crucial foundation for realistic synthetic data generation.
3
A Vision-Language-Kinematics (VLK) policy is trained on this synthetic data to predict short-horizon whole-body kinematic trajectories, enabling zero-shot transfer to a physical humanoid.

Conceptual Flow

HIGH LEVEL

Synthetic Data for Robot Learning

The robot learns by watching itself move and interact with objects in fake but realistic rooms, guided by simple instructions.

Real Room Scans

Robot Motion Data

Create Fake Practice World

Robot Sees

Robot Moves

Robot Hears

Humanoid Robot Performs Tasks

After practicing in fake rooms, the robot can successfully walk, turn, and move boxes in real rooms just by being told what to do.

Robot Sees

Robot Hears

Robot Decides & Acts

Walk to Spot

Pick Up Box

Put Down Box

This breakdown was generated by SciGrove. Get the same analysis — intuition, storyboard, peer review, a runnable prototype and a glossary — on any paper you upload or paste a DOI for.