Abstract Background Recent genome-wide association studies have identified genetic loci that jointly make a considerable contribution to risk of developing Alzheimer’s disease (AD). Because neuropathological features of AD can be present several decades before disease onset, we investigated whether effects of polygenic risk are detectable by neuroimaging in young adults. We hypothesised that higher polygenic risk scores (PRS) for AD would be associated with reduced volume of the hippocampus, and other limbic and paralimbic areas. We further hypothesised that AD PRS would affect the microstructure of fibre tracts connecting the hippocampus with other brain areas. Methods We analysed the association between AD PRS and brain imaging parameters using T1-weighted structural (n=272) and diffusion-weighted scans (n=197). Results We found a significant association between AD PRS and left hippocampal volume, with higher risk associated with lower left hippocampal volume (p=0.001). This effect remained when the APOE gene was excluded (p=0.031), suggesting that the relationship between hippocampal volume and AD is the result of multiple genetic factors, not exclusively variability in the APOE gene. The diffusion tensor imaging analysis revealed that fractional anisotropy of the right cingulum was inversely correlated with AD PRS (p=0.009). We thus show that polygenic effects of AD risk variants on brain structure can already be detected in young adults. Conclusions This finding paves the way for further investigation of the effects of AD risk variants and may become useful for efforts to combine genotypic and phenotypic data for risk prediction and to enrich future prevention trials of AD.