Obesity is a complex health issue in which the brain plays a role yet to be determined, especially in the elderly. Indeed, in the ageing population, the balance between fat and lean mass is different; thus, the co-influence between the brain and obesity may differ between the elderly and younger subjects. Our main goal is thus to explore the relationship between the brain and obesity using two different approaches to measure obesity: body mass index (BMI) and an index centred on fat mass, the body fat index (BFI). Among the 1,011 subjects of the PROOF population, 273 subjects aged 75 years underwent 3D magnetic resonance imaging as well as dual-energy X-ray absorptiometry to assess fat mass. Voxel-based morphometry was used to explore the local differences in brain volume with obesity. Higher BMI and BFI were associated with higher grey matter (GM) volume in the left cerebellum. Higher BMI and BFI were mainly associated with higher white matter volume in the left and right cerebellum and near the right medial orbital gyrus. Higher BMI was also associated with higher GM volume in the brainstem, whereas higher BFI was associated with higher GM volume in the left middle temporal gyrus. No decrease in white matter was associated with BMI or BFI. In the elderly, the relationship between the brain and obesity does not depend on the marker of obesity. Supra-tentorial brain structures seem to be slightly associated with obesity, whereas the cerebellum seems to be one of the key structures related to obesity.