SciGroveBeta
Neuroscience

Mapping Whisper Representations to Human ECoG Responses with Interpretable Time-Resolved Neural Encoding

Matteo Ciferri, Tommaso Boccato, Michal Olak, Matteo Ferrante, Nicola Toschi

Featured June 17, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new brain-reading model uses AI speech patterns to predict how the human brain processes sounds over time, showing that the AI's 'middle thoughts' best match brain activity and revealing how the brain organizes speech sounds.

In depth
The paper introduces a time-resolved neural encoder that maps internal representations from the Whisper speech model to human intracranial ECoG responses. This encoder combines speech embeddings with a recurrent temporal model and a soft attention mechanism to capture fine-grained, time-dependent stimulus-response relationships. The approach also includes a phonemic interpretability framework to identify anatomically coherent phoneme-category organization in cortical activity.

Key Takeaways

  • 1
    Intermediate layers of the Whisper speech model provide the strongest correspondence with human ECoG activity, supporting a hierarchical match between model representations and cortical speech processing.
  • 2
    The proposed time-resolved neural encoder, featuring a recurrent temporal model and soft attention, significantly outperforms linear baselines in predicting high-resolution ECoG responses during naturalistic speech perception.
  • 3
    A novel phonemic interpretability framework reveals anatomically coherent spatial clusters of phoneme-selective electrodes across the temporal cortex, linking model-derived features to known phonetic organization.

Conceptual Flow

HIGH LEVEL
1
Methodology: Mapping Speech to Brain Activity

The model takes spoken words, turns them into computer features, then uses a special 'attention' step to predict how the brain reacts to each part of the word.

Spoken Word
Convert to Features
Speech Features
Brain Activity Prediction
2
Results: Better Predictions and Brain Maps

The new method predicts brain activity better than older ones, especially showing how the brain processes speech sounds in specific areas.

Old Prediction Method
New Prediction Method
Compare Results
Less Accurate
More Accurate
Brain Sound Map