SciGroveBeta
Machine Learning

On-Policy Self-Distillation without Any Supervision

Yijiang Li, Bingyang Wang

Featured August 10, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

By having a language model generate many answers and then 'vote' on the best one, it can teach itself to fix its own mistakes without needing any outside help, making it smarter.

In depth
The paper introduces Unsupervised On-Policy Self-Distillation (U-OPSD), a novel method that enables large language models (LLMs) to improve their reasoning capabilities without any external supervision. It achieves this by leveraging the model's own internal consistency: multiple rollouts are sampled, and a pseudo-solution is constructed via majority vote. This pseudo-solution then serves as a privileged context for a teacher policy, which distills knowledge into the student policy by correcting its disagreeing rollouts.

Key Takeaways

  • 1
    Introduces U-OPSD, enabling LLMs to perform self-distillation without relying on external supervision like ground-truth labels or stronger teacher models.
  • 2
    Leverages internal consistency by sampling multiple rollouts and constructing a pseudo-solution via majority vote, which then guides the self-distillation process.
  • 3
    Demonstrates that distilling from a consensus-derived pseudo-solution into prefixes of the model's own incorrect completions can match or surpass supervised methods on mathematical reasoning benchmarks.

Conceptual Flow

HIGH LEVEL
1
Methodology: Self-Correction through Internal Consensus

The model generates many answers, picks the most common one as the 'right' answer, and then learns from its own wrong answers to get better.

Problem
Generate Many Answers Find Best Answer by Vote Teach Model to Fix Mistakes
Smarter Model
2
Results: Outperforming Supervised Methods Unsupervised

This new way of learning without help works just as well, or even better, than older methods that needed correct answers given to them.

Old Method (Needs Help)
New Method (No Help)
Compare Performance
New Method Wins