Machine Learning

CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru, MohammadHossein Rezaei, Aakash Sabharwal, Yunzhong He

Featured July 21, 2026

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new method called CRAFT helps large language models learn better by finding their exact weaknesses using detailed grading rules, then creating special practice data just for those weak spots.

In depth

The paper introduces CRAFT, a method that transforms rubric-based evaluations into a precise diagnosis of large language model (LLM) weaknesses. It achieves this by extracting specific capability descriptions from prompt-rubric pairs, organizing them into a hierarchical capability tree, and then using dynamic cross-level selection to identify low-performing nodes. These identified weak capabilities then directly guide the generation of targeted fine-tuning data, leading to improved model performance.

Key Takeaways

1
CRAFT provides a fine-grained diagnosis of LLM weaknesses by analyzing individual rubric criteria, offering a more precise understanding than traditional prompt- or category-level evaluations.
2
The method constructs a hierarchical capability tree from extracted rubric criteria and dynamically selects weak nodes at the most informative granularity, avoiding fixed-depth limitations.
3
Targeted fine-tuning data generated from CRAFT's diagnosis leads to measurably stronger models across professional domains (finance and legal) compared to untargeted or prompt-level approaches.

Conceptual Flow

HIGH LEVEL

Methodology: How CRAFT Works

Instead of just knowing a model is weak, this method finds *why* by breaking down grading rules into specific skills, organizing them like a family tree, and then making custom practice problems for the weak branches.

Grading Rules

Find Specific Skills

Organize Skills Like a Tree

Make Custom Practice

Results: Improved Model Performance

By using this smart way to find weaknesses and create practice, the models got better at tough tasks in finance and law compared to older methods.

Old Practice Method

CRAFT Practice Method

Train Models

Model Performance (Lower)

Model Performance (Higher)

This breakdown was generated by SciGrove. Get the same analysis — intuition, storyboard, peer review, a runnable prototype and a glossary — on any paper you upload or paste a DOI for.