Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word ...prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.
Brain dynamics are highly complex and yet hold the key to understanding brain function and dysfunction. The dynamics captured by resting-state functional magnetic resonance imaging data are noisy, ...high-dimensional, and not readily interpretable. The typical approach of reducing this data to low-dimensional features and focusing on the most predictive features comes with strong assumptions and can miss essential aspects of the underlying dynamics. In contrast, introspection of discriminatively trained deep learning models may uncover disorder-relevant elements of the signal at the level of individual time points and spatial locations. Yet, the difficulty of reliable training on high-dimensional low sample size datasets and the unclear relevance of the resulting predictive markers prevent the widespread use of deep learning in functional neuroimaging. In this work, we introduce a deep learning framework to learn from high-dimensional dynamical data while maintaining stable, ecologically valid interpretations. Results successfully demonstrate that the proposed framework enables learning the dynamics of resting-state fMRI directly from small data and capturing compact, stable interpretations of features predictive of function and dysfunction.
We present open-source tools for three-dimensional (3D) analysis of photographs of dissected slices of human brains, which are routinely acquired in brain banks but seldom used for quantitative ...analysis. Our tools can: (1) 3D reconstruct a volume from the photographs and, optionally, a surface scan; and (2) produce a high-resolution 3D segmentation into 11 brain regions per hemisphere (22 in total), independently of the slice thickness. Our tools can be used as a substitute for ex vivo magnetic resonance imaging (MRI), which requires access to an MRI scanner, ex vivo scanning expertise, and considerable financial resources. We tested our tools on synthetic and real data from two NIH Alzheimer’s Disease Research Centers. The results show that our methodology yields accurate 3D reconstructions, segmentations, and volumetric measurements that are highly correlated to those from MRI. Our method also detects expected differences between post mortem confirmed Alzheimer’s disease cases and controls. The tools are available in our widespread neuroimaging suite ‘FreeSurfer’ ( https://surfer.nmr.mgh.harvard.edu/fswiki/PhotoTools ).
Every year, thousands of human brains are donated to science. These brains are used to study normal aging, as well as neurological diseases like Alzheimer’s or Parkinson’s. Donated brains usually go to ‘brain banks’, institutions where the brains are dissected to extract tissues relevant to different diseases. During this process, it is routine to take photographs of brain slices for archiving purposes. Often, studies of dead brains rely on qualitative observations, such as ‘the hippocampus displays some atrophy’, rather than concrete ‘numerical’ measurements. This is because the gold standard to take three-dimensional measurements of the brain is magnetic resonance imaging (MRI), which is an expensive technique that requires high expertise – especially with dead brains. The lack of quantitative data means it is not always straightforward to study certain conditions. To bridge this gap, Gazula et al. have developed an openly available software that can build three-dimensional reconstructions of dead brains based on photographs of brain slices. The software can also use machine learning methods to automatically extract different brain regions from the three-dimensional reconstructions and measure their size. These data can be used to take precise quantitative measurements that can be used to better describe how different conditions lead to changes in the brain, such as atrophy (reduced volume of one or more brain regions). The researchers assessed the accuracy of the method in two ways. First, they digitally sliced MRI-scanned brains and used the software to compute the sizes of different structures based on these synthetic data, comparing the results to the known sizes. Second, they used brains for which both MRI data and dissection photographs existed and compared the measurements taken by the software to the measurements obtained with MRI images. Gazula et al. show that, as long as the photographs satisfy some basic conditions, they can provide good estimates of the sizes of many brain structures. The tools developed by Gazula et al. are publicly available as part of FreeSurfer, a widespread neuroimaging software that can be used by any researcher working at a brain bank. This will allow brain banks to obtain accurate measurements of dead brains, allowing them to cheaply perform quantitative studies of brain structures, which could lead to new findings relating to neurodegenerative diseases.
Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic ...representations posited by traditional psycholinguistics. We hypothesize that language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language. To test this hypothesis, we densely record the neural activity patterns in the inferior frontal gyrus (IFG) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derive a continuous vectorial representation for each word (i.e., a brain embedding) in each patient. Using stringent zero-shot mapping we demonstrate that brain embeddings in the IFG and the DLM contextual embedding space have common geometric patterns. The common geometric patterns allow us to predict the brain embedding in IFG of a given left-out word based solely on its geometrical relationship to other non-overlapping words in the podcast. Furthermore, we show that contextual embeddings capture the geometry of IFG embeddings better than static word embeddings. The continuous brain embedding space exposes a vector-based neural code for natural language processing in the human brain.
There has been an upward trend in developing frameworks that enable neuroimaging researchers to address challenging questions by leveraging data across multiple sites all over the world. One such ...open-source framework is the Collaborative Informatics and Neuroimaging Suite Toolkit for Anonymous Computation (COINSTAC) that works on Windows, macOS, and Linux operating systems and leverages containerized analysis pipelines to analyze neuroimaging data stored locally across multiple physical locations without the need for pooling the data at any point during the analysis. In this paper, the COINSTAC team partnered with a data collection consortium to implement the first-ever decentralized voxelwise analysis of brain imaging data performed outside the COINSTAC development group. Decentralized voxel-based morphometry analysis of over 2000 structural magnetic resonance imaging data sets collected at 14 different sites across two cohorts and co-located in different countries was performed to study the structural changes in brain gray matter which linked to age, body mass index (BMI), and smoking. Results produced by the decentralized analysis were consistent with and extended previous findings in the literature. In particular, a widespread cortical gray matter reduction (resembling a ‘default mode network’ pattern) and hippocampal increase with age, bilateral increases in the hypothalamus and basal ganglia with BMI, and cingulate and thalamic decreases with smoking. This work provides a critical real-world test of the COINSTAC framework in a “Large-N” study. It showcases the potential benefits of performing multivoxel and multivariate analyses of large-scale neuroimaging data located at multiple sites.
Collaborative neuroimaging research is often hindered by technological, policy, administrative, and methodological barriers, despite the abundance of available data. COINSTAC (The Collaborative ...Informatics and Neuroimaging Suite Toolkit for Anonymous Computation) is a platform that successfully tackles these challenges through federated analysis, allowing researchers to analyze datasets without publicly sharing their data. This paper presents a significant enhancement to the COINSTAC platform: COINSTAC Vaults (CVs). CVs are designed to further reduce barriers by hosting standardized, persistent, and highly-available datasets, while seamlessly integrating with COINSTAC's federated analysis capabilities. CVs offer a user-friendly interface for self-service analysis, streamlining collaboration, and eliminating the need for manual coordination with data owners. Importantly, CVs can also be used in conjunction with open data as well, by simply creating a CV hosting the open data one would like to include in the analysis, thus filling an important gap in the data sharing ecosystem. We demonstrate the impact of CVs through several functional and structural neuroimaging studies utilizing federated analysis showcasing their potential to improve the reproducibility of research and increase sample sizes in neuroimaging studies.
In the field of neuroimaging, there is a growing interest in developing collaborative frameworks that enable researchers to address challenging questions about the human brain by leveraging data ...across multiple sites all over the world. Additionally, efforts are also being directed at developing algorithms that enable collaborative analysis and feature learning from multiple sites without requiring the often large data to be centrally located. In this paper, we propose two new decentralized algorithms: (1) A decentralized regression algorithm for performing a voxel-based morphometry analysis on structural magnetic resonance imaging (MRI) data and, (2) A decentralized dynamic functional network connectivity algorithm which includes decentralized group ICA and sliding-window analysis of functional MRI data. We compare results against those obtained from their pooled (or centralized) counterparts on the same data i.e., as if they are at one site. Results produced by the decentralized algorithms are similar to the pooled-case and showcase the potential of performing multi-voxel and multivariate analyses of data located at multiple sites. Such approaches enable many more collaborative and comparative analysis in the context of large-scale neuroimaging studies.
Abstract
Deep learning has become an effective tool for classifying biological sex based on functional magnetic resonance imaging (fMRI). However, research on what features within the brain are most ...relevant to this classification is still lacking. Model interpretability has become a powerful way to understand “black box” deep-learning models, and select features within the input data that are most relevant to the correct classification. However, very little work has been done employing these methods to understand the relationship between the temporal dimension of functional imaging signals and the classification of biological sex. Consequently, less attention has been paid to rectifying problems and limitations associated with feature explanation models, e.g. underspecification and instability. In this work, we first provide a methodology to limit the impact of underspecification on the stability of the measured feature importance. Then, using intrinsic connectivity networks from fMRI data, we provide a deep exploration of sex differences among functional brain networks. We report numerous conclusions, including activity differences in the visual and cognitive domains and major connectivity differences.
Background
In Alzheimer’s Disease (AD) and other types of dementia, neuropathological assessment serves as ground truth for diagnosis and validation of novel biomarkers. Such assessment commonly ...implies inspection of a single histological slide per region of interest. Pathology burden quantification, as well as investigation of its spatial distribution, is thus biased by the variability of tissue sampling in routinary neuropathological practice. Numerous efforts have sought to overcome this lack of generalizability using 3D reconstruction of histology, but available approaches rely on dense sectioning and an MRI scan, which may not be attainable by many brain banks.
Method
We present a histology mapping pipeline that uses blockface images of brain hemisphere slices for 3D spatial localization. The slices are first geometrically stacked and coarsely aligned using a probabilistic atlas. Next, deep learning contrast synthesis is employed to obtain a high‐resolution 3D image of the hemisphere with MRI‐like contrast, which is subsequently registered nonlinearly to a standard atlas (MNI). Histological slides are then registered to the corresponding coronal slice within the subject‐specific reconstructed hemisphere. Finally, the registrations are concatenated to deform the histology to the standard space. We show the applicability of this pipeline by analyzing the distribution of tau AD pathology along the longitudinal axis of the hippocampus in 26 patients with no significant co‐pathologies aside from AD.
Result
We took advantage of the inter‐subject spatial variability observed after 3D slide mapping to achieve extensive sampling along the hippocampal long axis. Our results show an anterior‐posterior gradient of tau burden in the hippocampus that is consistent with previous work using ex vivo MRI and dense histological sampling, further supporting the reliability of the pipeline.
Conclusion
Our pipeline enables studying pathology in 3D using sparse sampling and in absence of a subject‐specific MRI reference. Moreover, given that dissection photography is standard practice in brain banks, our method is widely applicable to any prospective or retrospective human histology dataset.
Acute ischaemic stroke disturbs healthy brain organization, prompting subsequent plasticity and reorganization to compensate for the loss of specialized neural tissue and function. Static resting ...state functional MRI studies have already furthered our understanding of cerebral reorganization by estimating stroke-induced changes in network connectivity aggregated over the duration of several minutes. In this study, we used dynamic resting state functional MRI analyses to increase temporal resolution to seconds and explore transient configurations of motor network connectivity in acute stroke. To this end, we collected resting state functional MRI data of 31 patients with acute ischaemic stroke and 17 age-matched healthy control subjects. Stroke patients presented with moderate to severe hand motor deficits. By estimating dynamic functional connectivity within a sliding window framework, we identified three distinct connectivity configurations of motor-related networks. Motor networks were organized into three regional domains, i.e. a cortical, subcortical and cerebellar domain. The dynamic connectivity patterns of stroke patients diverged from those of healthy controls depending on the severity of the initial motor impairment. Moderately affected patients (n = 18) spent significantly more time in a weakly connected configuration that was characterized by low levels of connectivity, both locally as well as between distant regions. In contrast, severely affected patients (n = 13) showed a significant preference for transitions into a spatially segregated connectivity configuration. This configuration featured particularly high levels of local connectivity within the three regional domains as well as anti-correlated connectivity between distant networks across domains. A third connectivity configuration represented an intermediate connectivity pattern compared to the preceding two, and predominantly encompassed decreased interhemispheric connectivity between cortical motor networks independent of individual deficit severity. Alterations within this third configuration thus closely resembled previously reported ones originating from static resting state functional MRI studies post-stroke. In summary, acute ischaemic stroke not only prompted changes in connectivity between distinct networks, but it also caused characteristic changes in temporal properties of large-scale network interactions depending on the severity of the individual deficit. These findings offer new vistas on the dynamic neural mechanisms underlying acute neurological symptoms, cortical reorganization and treatment effects in stroke patients.
PDF
