Cross-modal processing of auditory and visual symbol representations in the temporo-parietal cortex

Numeracy and literacy are fundamental cognitive skills that rely on associating visual symbols with their spoken representations. Prior research has identified the posterior temporal-parietal cortex as a key neural region for the cross-modal transformation of these audio-visual alphanumeric symbols. However, the modality-dependent and cross-modal cortical activation patterns underlying these transformations remain unclear.

In this slow-event-related 3T fMRI experiment, twenty-one participants were presented with auditory or visual letters and numbers while performing a passive listening/viewing task. We found overlapping activation across auditory cortical regions for auditory letters/numbers and across ventral visual regions for visual letters/numbers. In particular, activity in superior temporal cortical regions such as A5/A4/Parabelt exhibited high reliability for auditory stimuli, whereas activity in occipital and ventral temporal cortical regions such as V3/V4/PH demonstrated high reliability for visual stimuli.

The temporo-parieto-occipital junction (TPOJ) showed overlapping responses with similar amplitudes for both auditory and visual stimuli. Despite this global similarity in responses, multivariate analysis revealed that the right TPOJ successfully differentiated between visual and auditory stimuli. Our findings reinforce the TPOJ’s role in the cross-modal processing of symbolic representations and may have implications for developmental learning difficulties such as dyslexia, where cross-modal integration may form a challenge for acquiring reading fluency.

We thank Daniel Ansari for his advice during the conceptualization of the experiment and Agustin Lage-Castellanos for his involvement in the initial analysis steps. We thank all the participants for their involvement in this study. This work was supported by Maastricht Brain Imaging Center (MBIC) funding, the Netherlands Organization for Scientific Research (NWO, Vidi 452-16-004 and Vici #VI.C.221.025 to MB), and the China Scholarship Council (CSC 202107720051 to ZC).

Maastricht Brain Imaging Center, Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, Netherlands Zhiwei Chen, Jan W. Dowdle, Francesco Gentile & Milene Bonte Department of Psychological and Social Sciences, John Cabot University, Rome, Italy Donders Institute for Brain, Cognition and Behaviour Radboud University, Nijmegen, Netherlands The authors declare no competing interests. Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material.