Machine Learning

Weak-to-Strong Generalization via Direct On-Policy Distillation

Shiyuan Feng, Huan-ang Gao, et al.

Featured July 19, 2026

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

Instead of expensive training on big language models, this paper shows how to teach a strong model by showing it *how* a small model improved with practice, using the difference in its behavior before and after learning as a secret hint.

In depth

The paper introduces Direct On-Policy Distillation (Direct-OPD), a method to efficiently transfer reasoning capabilities from a smaller, weaker language model (LM) to a stronger one. Instead of directly imitating the weak teacher's final policy, which can be limited by its capacity, Direct-OPD distills the policy shift induced by reinforcement learning (RL) on the weak model. This shift, calculated as the log-ratio between the post-RL teacher and its pre-RL reference, acts as a dense implicit reward, allowing stronger students to learn the *direction* of improvement without incurring the high computational cost of running RL directly on large models.

Key Takeaways

1
Direct-OPD transfers the RL-induced policy shift (log-ratio of post-RL vs. pre-RL weak teacher) as an implicit reward, rather than imitating the weak teacher's final policy.
2
This approach enables weak-to-strong generalization, improving stronger target models at a fraction of the computational cost compared to direct RL on large models.
3
The method consistently improves various student models, even those initially stronger than the teacher, and allows for sequential composition of multiple policy shifts.

Conceptual Flow

HIGH LEVEL

Teaching Strong Models Efficiently

Instead of directly copying a small, improved model, the method teaches a big model by showing it *how* the small model changed its mind after learning.

Small Model Before Learning

Small Model After Learning

Find Behavior Change

Learning Direction for Big Model

Faster, Better Learning for Big Models

This new way of teaching makes big models learn much faster and better than traditional methods, even when the small teacher isn't as smart.

Big Model (Initial)

Learning Direction

Apply Smart Hints

Big Model (Improved, Fast)