SciGroveBeta
Machine Learning

CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru, MohammadHossein Rezaei, Aakash Sabharwal, Yunzhong He

Featured July 21, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new method called CRAFT helps large language models learn better by finding their exact weaknesses using detailed grading rules, then creating special practice data just for those weak spots.

In depth
The paper introduces CRAFT, a method that transforms rubric-based evaluations into a precise diagnosis of large language model (LLM) weaknesses. It achieves this by extracting specific capability descriptions from prompt-rubric pairs, organizing them into a hierarchical capability tree, and then using dynamic cross-level selection to identify low-performing nodes. These identified weak capabilities then directly guide the generation of targeted fine-tuning data, leading to improved model performance.

Key Takeaways

  • 1
    CRAFT provides a fine-grained diagnosis of LLM weaknesses by analyzing individual rubric criteria, offering a more precise understanding than traditional prompt- or category-level evaluations.
  • 2
    The method constructs a hierarchical capability tree from extracted rubric criteria and dynamically selects weak nodes at the most informative granularity, avoiding fixed-depth limitations.
  • 3
    Targeted fine-tuning data generated from CRAFT's diagnosis leads to measurably stronger models across professional domains (finance and legal) compared to untargeted or prompt-level approaches.

Conceptual Flow

HIGH LEVEL
1
Methodology: How CRAFT Works

Instead of just knowing a model is weak, this method finds *why* by breaking down grading rules into specific skills, organizing them like a family tree, and then making custom practice problems for the weak branches.

Grading Rules
Find Specific Skills
Organize Skills Like a Tree
Make Custom Practice
2
Results: Improved Model Performance

By using this smart way to find weaknesses and create practice, the models got better at tough tasks in finance and law compared to older methods.

Old Practice Method
CRAFT Practice Method
Train Models
Model Performance (Lower)
Model Performance (Higher)