SciGroveBeta
Robotics

Point Tracking Improves World Action Models

Jiarui Guan, Wenshuai Zhao, Yue Pei, Ziliang Chen, Arno Solin, Juho Kannala

Featured May 27, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

Robots learn better by predicting not just future pictures, but also where key points on objects will move and if they'll be hidden, using a joint pixel-and-track model to make actions more reliable.

In depth
The paper introduces JOPAT, a JOint Pixel-And-Track World-Action Model designed to enhance robot policy learning by explicitly modeling motion. It addresses the limitation of pixel-only world models, which struggle with task-irrelevant visual variations and implicit motion encoding, by jointly predicting future visual latents, 2D point tracks with visibility, and robot actions within a single denoising diffusion transformer. This approach provides a more robust and explicit representation of long-horizon dynamics, especially under occlusion or out-of-frame motion, leading to improved manipulation performance.

Key Takeaways

  • 1
    JOPAT integrates visual latents, 2D point tracks with visibility, and robot actions into a unified generative framework, providing an explicit correspondence-level motion interface for robot control.
  • 2
    The model achieves state-of-the-art performance on 40 LIBERO manipulation tasks, demonstrating strong robustness under visual distribution shifts, occlusions, and long-horizon dependencies.
  • 3
    Explicit point-track modeling is crucial for robust performance, improving long-horizon consistency and mitigating occlusion effects by providing stable motion grounding that complements pixel-based representations.

Conceptual Flow

HIGH LEVEL
1
Jointly Predicting Future States

The robot sees what's happening, then guesses what the world will look like, where important points will move, and what action it should take, all at the same time.

Current View
Robot Action Idea
Guess Future
Future View
Future Point Paths
Final Robot Action
2
Better Robot Control

By guessing future pictures and point movements together, the robot can do tricky tasks much better, especially when things get hidden or move far.

Old Robot Method
New Robot Method
Compare Performance
Less Reliable Control
More Reliable Control