Medicine

EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

Yitong Qiao, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu, Zhixuan Chu, Kui Ren

Featured July 12, 2026

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new test called EHR-Complex helps smart computer programs learn to answer tricky medical questions by letting them try out database commands and learn from their mistakes, just like a doctor would.

In depth

The paper introduces EHR-Complex, a novel large-scale benchmark designed to rigorously evaluate clinical agents on complex, interactive electronic health record (EHR) reasoning tasks. Unlike prior static text-to-SQL benchmarks, EHR-Complex requires agents to execute SQL queries or Python code in a sandboxed environment, providing real-time feedback and reflecting the compositional and longitudinal nature of real-world EHR analysis. This interactive setup, built on the full MIMIC-IV database, exposes critical limitations in current large language models, particularly in handling multi-table aggregation and medical-code grounding.

Key Takeaways

1
EHR-Complex is a new benchmark for interactive clinical database reasoning, built on the full MIMIC-IV dataset with 52K tasks.
2
It features real-world SQL complexity, averaging 31.93 structural components per query, requiring multi-table aggregation and compositional reasoning.
3
Evaluations reveal significant gaps in current LLM agents, especially for population-level reasoning and medical-code grounding, with the best model achieving only 62.3% accuracy.

Conceptual Flow

HIGH LEVEL

Methodology: How EHR-Complex is Built

The paper built a huge medical question-answering test by turning patient records into event maps, then finding common medical stories, and finally making computer code questions from them.

Patient Records

Create Event Maps

Patient Event Graphs

Results: What They Found

Even the smartest computer programs struggled with the new medical questions, especially when asked about many patients at once, showing they still need to get much better.

Smart Programs

Try New Questions

Low Success Rate

This breakdown was generated by SciGrove. Get the same analysis — intuition, storyboard, peer review, a runnable prototype and a glossary — on any paper you upload or paste a DOI for.