S-127 | Temporal Correspondence Between Human Brain and Vision Transformers in Rapid Stimuli Tasks

Theoretical and Computational Neuroscience
Author: Giovanni Franco Gabriel Marraffini | Email: giovanni.marraffini@gmail.com

Giovanni Franco Gabriel Marraffini, Eric Lützow holm^1°2°, Enzo Tagliazucchi^1°2°3°

^1° Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Godoy Cruz 2290, CABA 1425, Argentina.
^2° Instituto de Física Interdisciplinaria y Aplicada, Departamento de Física, Universidad de Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina.
^3° Latin American Brain Health (BrainLat), Universidad Adolfo Ibáñez, Av. Diag. Las Torres 2640, Peñalolén 7941169, Santiago Región Metropolitana, Chile

Visual processing in the human cerebral cortex is hierarchical and highly parallelized. Similarly, in the deep learning architecture known as vision transformers, data is processed iteratively through attention operations, which allow for the parallel evaluation of the importance of certain aspects of the image in the context of the rest of the image. In both the human brain and transformers, the attention mechanism selects parts of the input that are more relevant to the task at hand, enabling the model to prioritize and weigh this information during processing. In this work, we investigate the temporal correspondence between both systems, based on the hypothesis that each successive layer of the transformer will be correlated with temporally ordered components of the EEG response in an object recognition task. First, we analyze the representational similarity between the two systems, then we investigate predicting brain activations based on neural network activations, and finally, we use a state-of-the-art metric to make comparisons between neural network representations. We find that the early layers of the network are mostly aligned with the early stages of electrophysiological signals, while the advanced layers correspond to later stages.