SciGroveBeta
Machine Learning

ICRL: Learning to Internalize Self-Critique with Reinforcement Learning

Jianbo Lin, Xiaomin Yu, Yi Xin, Yifu Guo, Zhuosong Jiang, Zhongqi Yue, Weishi Wang, Heqing Zou, Chengwei Qin, Hui Xiong

Featured May 21, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new method teaches AI agents to truly learn from their mistakes, not just fix them when told, by having a solver and critic learn together and carefully transferring useful feedback into the agent's core skills.

In depth
The paper introduces ICRL, a novel framework that jointly trains a solver and a critic from a shared backbone to convert critique-induced success into unassisted solver ability. It addresses the challenge of critique internalization by introducing a distribution-calibration re-weighting ratio to correct for distributional shifts and employs role-wise group advantage estimation to stabilize joint optimization, ensuring the solver learns to improve without external critique.

Key Takeaways

  • 1
    ICRL jointly trains a solver and a critic from a shared backbone, enabling iterative self-improvement and the internalization of critique into the solver's unassisted capabilities.
  • 2
    A distribution-calibration re-weighting ratio is introduced to correct the distributional shift between critique-conditioned and critique-free behavior, ensuring the solver learns patterns compatible with its own prompt distribution.
  • 3
    Role-wise group advantage estimation stabilizes the joint optimization of the solver and critic by normalizing their distinct reward signals separately, preserving effective learning for each role.

Conceptual Flow

HIGH LEVEL
1
Methodology (The 'Logic')

The system uses two AI parts, a problem-solver and a mistake-finder, that learn together to make the solver better on its own.

Problem
Solver Tries
Fails
2
Results (The 'Impact')

This new way helps AI agents solve problems much better, especially hard math and complex tasks, even outperforming bigger AI models.

Old AI Method
Solves Problems
Some Success