Adverse childhood events may have differential effects on the brain that persist into adulthood. Findings on structural brain alterations in older adults exposed to early-life adversity are inconsistent notably due to heterogeneity in imaging studies, population, psychiatric comorbidities, nature of adverse events, and genetic vulnerability. This study examines whether exposure related to physical or sexual maltreatment, emotional maltreatment, and global adverse environment during childhood are associated with specific alterations in grey matter volumes and if this varies according to sex and serotonin transporter-linked promoter region (5-HTTLPR) genotype. Structural MRI was used to acquire anatomical scans from 398 community-dwelling older adults. Quantitative regional estimates of 23 subregional volumes were derived using FreeSurfer software. Retrospective reporting of childhood adversity was collected using structured self-reported questionnaire. Analyses adjusted for age, sex, brain volume, head injury, lifetime depression and anxiety disorder, psychiatric medication, and cardiovascular ischemic pathologies. Exposure to adverse family environment was associated with smaller volumes of several frontal, cingulate, and parietal subregions and larger amygdala in the 5-HTTLPR SS genotype participants specifically but larger volumes of caudate, putamen, pallidum, and nucleus accumbens in the SL genotype participants. Highly significant differences were found with excessive sharing of parent problems with children, associated with larger grey-matter volumes in the thalamus and several frontal and parietal regions in 5-HTTLPR SL male participants specifically. Early-life adversity is associated with grey-matter volume alterations in older adults and this varies according to the type of adversity experienced, sex, and serotonergic genetic vulnerability; 5-HTTLPR SS participants appearing most vulnerable and SL individuals most resilient. •Early adverse events are associated with brain structural alterations in later life.•Their effects differ according to type of adversity, gender, and 5-HTTLPR genotype.•5-HTTLPR SS individuals appear most vulnerable and SL individuals most resilient.