The aim of this study was to disentangle the effects of various genetic factors on hippocampal subfield volumes using three different approaches: a biologically driven candidate gene approach, a hypothesis-free GWAS approach, and a polygenic approach, where AD risk alleles are combined with a polygenic risk score (PRS). The impact of these genetic factors was investigated in a large dementia-free general population cohort from the Study of Health in Pomerania (SHIP, = 1806). Analyses were performed using linear regression models adjusted for biological and environmental risk factors. Hippocampus subfield volume alterations were found for ε4, Val, and L allele carriers. In addition, we were able to replicate GWAS findings, especially for rs17178139 ( ), rs1861979 ( ), rs7873551 ( ), and rs572246240 ( ). Interaction analyses between the significant SNPs as well as the PRS for AD revealed no significant results. Our results confirm that hippocampal volume reductions are influenced by genetic variation, and that different variants reveal different association patterns that can be linked to biological processes in neurodegeneration. Thus, this study underlines the importance of specific genetic analyses in the quest for acquiring deeper insights into the biology of hippocampal volume loss, memory impairment, depression, and neurodegenerative diseases.