SciGroveBeta
Machine Learning

How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size

Fabian Schaipp

Featured July 7, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new formula helps predict how well big AI models will learn by looking at their size, how many examples they see at once (batch size), and how many times they learn (training steps), even with less data for testing.

In depth
The paper introduces a three-term scaling law that models the test loss of large language models as a function of model size (), batch size (), and training steps (). This innovation allows for robust prediction of optimal batch size even with limited training data, and describes performance in suboptimal batch size regimes, bridging empirical scaling with optimization theory.

Key Takeaways

  • 1
    A novel three-term scaling law explicitly models loss based on model size, batch size, and training steps, unifying Chinchilla-type and hyperparameter scaling laws.
  • 2
    The proposed law enables robust fitting of optimal batch size scaling with significantly fewer training runs compared to prior methods, leading to substantial compute savings.
  • 3
    It accurately describes critical batch size phenomena and allows for modeling performance under suboptimal batch sizes, which is crucial for practical hardware constraints.

Conceptual Flow

HIGH LEVEL
1
Methodology: How the new scaling law works

Instead of just model size and total data, this method adds batch size and training steps to predict how well an AI learns, like knowing not just how much food you have, but how big each bite is and how many bites you take.

Model Size
Batch Size
Training Steps
Combine & Predict
Predicted Learning Error
2
Results: What did they find?

This new way helps find the best 'bite size' for learning much faster, saving a lot of computer power, and also shows how well the AI will do even if the 'bite size' isn't perfect.

Less Test Data
New Formula
Find Best Settings
Optimal Batch Size
Predict Performance