Materials

OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields

Wanhao Liu, Jiaqing Xie, Qian Tan, Weida Wang, Jue Wang, Ran Sun, Zhuo Yang, Wanli Ouyang, Lei Bai, TianFan Fu, Lu Chen, Xin Chen, Yuqiang Li

Featured June 24, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new science test called OmniMatBench helps figure out if smart computer programs can truly think like materials scientists, showing they're good at talking but often struggle with real calculations and problem-solving.

In depth

The paper introduces OmniMatBench, a novel benchmark designed to rigorously assess multimodal language models' (MLLMs) reasoning capabilities in materials science. It features 3,171 expert-curated problems spanning 19 subfields, moving beyond simple knowledge recall to evaluate complex scientific execution, formula application, and multimodal interpretation. The fine-grained evaluation protocol pinpoints specific failure modes, revealing a significant gap between MLLM fluency and reliable scientific reasoning.

Key Takeaways

1
OmniMatBench is a new, comprehensive benchmark for materials science, featuring 3,171 expert-curated problems across 19 subfields, designed to test multimodal reasoning beyond simple knowledge recall.
2
The benchmark employs a fine-grained evaluation protocol for both QA and calculation tasks, assessing not just final answers but also reasoning paths, formula selection, unit awareness, and output formatting.
3
Evaluations of 13 MLLMs reveal a substantial knowledge-to-execution gap, with even top models struggling with specialized engineering scenarios, formula application, and visual-parameter grounding.

Conceptual Flow

HIGH LEVEL

Methodology (The "Logic")

Experts gather science problems, check them carefully, and then use them to create a tough test for smart computer programs.

Old Science Books

Expert Scientists

Create New Problems

Tough Science Test

Results (The "Impact")

Even the smartest computer programs struggle with the new science test, showing they still need to learn how to solve real-world materials problems.

Smart Computer Programs

Tough Science Test

Try to Solve

Low Scores

Many Mistakes