SciGroveBeta
Machine Learning

Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models

Antyabha Rahman, Akshaj Gurugubelli, Omar Ankit, Kevin Zhu, Aishwarya Balwani

Featured August 7, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

Smart math models trained with reinforcement learning learn to "see" correct answers much more clearly inside their brains, making deeper layers super important for solving problems.

In depth
The study reveals that Reinforcement Learning (RL) models for mathematical reasoning develop clearer, more linearly separable internal representations of correctness, which emerge earlier in the network. This is coupled with a hierarchical computational architecture where deeper layers are progressively more critical, unlike Supervised Fine-Tuned (SFT) models that distribute importance uniformly.

Key Takeaways

  • 1
    RL models encode correctness more clearly: They develop linearly separable representations of correct answers earlier in the network, detectable by linear probes.
  • 2
    RL training creates hierarchical architectures: Deeper layers in RL models become progressively more critical for reasoning, contrasting with the uniform importance distribution in SFT models.
  • 3
    Token allocation varies by model: While some RL models show higher token variability, others maintain consistent low variability, suggesting pipeline-specific influences beyond just RL vs. SFT.

Conceptual Flow

HIGH LEVEL
1
Investigating Model 'Brain' Differences

The researchers looked inside the models using three main tools to see how they think differently.

Model's Internal Thoughts
Analyze with 3 Tools
Clarity of Ideas
Important Thinking Steps
Consistency of Output
2
Key Discoveries About RL Models

They found that models trained with rewards have clearer internal ideas and use their deeper thinking parts more.

RL Model Training
SFT Model Training
Leads To
Clearer Internal Ideas (RL)
Deeper Layers More Critical (RL)
Output Consistency Varies