SciGroveBeta
Medicine

Towards Conversational Medical AI with Eyes, Ears and a Voice

Meet Shah, Jason Gusdorf

Featured June 5, 2026

Original Paper

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new AI doctor's assistant uses live video and audio to understand patients like a human, splitting its brain into a 'Talker' for chat and a 'Planner' for smart medical thinking, helping doctors make better decisions.

In depth
The paper introduces "AI co-clinician," a novel conversational AI system for telemedicine that processes continuous audio-visual data in real-time. Its dual-agent architecture, comprising a 'Talker' for low-latency interaction and a 'Clinical Planner' for deep reasoning, enables it to balance empathetic dialogue with rigorous diagnostic capabilities, addressing the limitations of text-only medical AIs in capturing nuanced patient cues and performing guided physical examinations.

Key Takeaways

  • 1
    The dual-agent architecture (Talker and Clinical Planner) is a key innovation, allowing the AI to maintain conversational fluency while performing deep clinical reasoning in real-time.
  • 2
    The system integrates continuous audio-visual streams from live patient conversations, enabling it to interpret non-verbal cues and guide physical examinations, a significant advancement over prior text-only or static multimodal approaches.
  • 3
    A novel evaluation framework, combining universal TelePACES and case-specific rubrics, was developed to rigorously assess real-time audio-visual clinical encounters, revealing unique failure modes like "contextual completion."

Conceptual Flow

HIGH LEVEL
1
Methodology: How the AI Thinks and Talks

The AI uses two parts: one talks and sees quickly, the other thinks deeply about medical rules, so it can be fast and smart at the same time.

Patient Video
Patient Audio
AI Brain Works
AI Response
2
Results: How Well the AI Performed

The new AI performed much better than other AIs and almost as well as human doctors in many areas, especially in understanding visual clues.

Old AI Performance
Compare Results
New AI Performance
Human Doctor Performance

This breakdown was generated by SciGrove. Get the same analysis — intuition, storyboard, peer review, a runnable prototype and a glossary — on any paper you upload or paste a DOI for.