The impact of prediabetes and diabetes on cognitive decline and the potential underlying mechanisms remain unclear. We investigated whether prediabetes and diabetes accelerate cognitive decline and brain aging, and the initial pathological changes linked to microvascular processes. Nine-year longitudinal data from the Swedish National Study on Aging and Care-Kungsholmen (n = 2746, age ≥60 years) and the magnetic resonance imaging subsample (n = 455) were used. Cognitive function was assessed with Mini-Mental State Examination. Brain magnetic resonance imaging markers included total brain tissue, white matter, gray matter, white matter hyperintensities, and hippocampal volumes. Compared with diabetes-free status, prediabetes and diabetes were independently associated with accelerated cognitive decline. Prediabetes was cross-sectionally associated with smaller total brain tissue volume (P < .01), particularly smaller white matter volume. Diabetes was associated with larger white matter hyperintensities volume. Longitudinally, diabetes was associated with faster white matter hyperintensities accumulation. No associations between prediabetes or diabetes and hippocampal volume were found. Diabetes and prediabetes accelerate cognitive decline and might predict microvascular lesions among dementia-free older adults.