Robotics

Recovering Hidden Reward in Diffusion-Based Policies

Yanbiao Ji, Qiuchang Li, Yuting Hu, Shaokai Wu, Wenyuan Xie, Guodong Zhang, Qicheng He, Deyi Ji, Yue Ding, Hongtao Lu

Featured May 19, 2026

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

By learning a special 'energy map' where good actions have low energy, the paper's method can both generate smart robot actions and figure out *why* those actions are good, making robots more robust.

In depth

The paper introduces ENERGY FLOW, a framework that unifies generative action modeling with inverse reinforcement learning. It parameterizes a scalar energy function whose gradient directly serves as the denoising field for action generation. Crucially, the authors prove that this learned score function also recovers the gradient of the expert's soft Q-function, enabling reward extraction without complex adversarial training and improving out-of-distribution generalization through a conservative field constraint.

Key Takeaways

1
The ENERGY FLOW framework unifies diffusion-based policy learning and inverse reinforcement learning by parameterizing a scalar energy function whose gradient is the denoising field.
2
The authors theoretically prove that the learned score function (gradient of the energy) is proportional to the gradient of the expert's soft Q-function, allowing for direct reward extraction.
3
Enforcing a conservative field constraint on the energy function acts as a powerful inductive bias, reducing hypothesis complexity and significantly improving out-of-distribution generalization and robustness.

Conceptual Flow

HIGH LEVEL

Methodology: Learning an Energy Landscape

The method learns a hidden 'energy map' from expert actions, which then helps both create new actions and understand what makes them good.

Expert Actions

Learn Energy Map

Action Generator

Reward Signal

Results: Robust Actions and Clear Rewards

By adding a 'smoothness rule' to the energy map, the new method creates better actions and provides a clear reward signal, outperforming older methods.

Old Action Policy

No Clear Reward

Add Energy Constraint

Better Actions

Clear Reward

This breakdown was generated by SciGrove. Get the same analysis — intuition, storyboard, peer review, a runnable prototype and a glossary — on any paper you upload or paste a DOI for.