Objectives To examine associations between adiposity and adiposity change (loss, stable, gain) and subsequent longitudinal cognitive performance in African Americans in mid and late life. Design Cohort study using linear mixed models. Setting Genetic Epidemiology Network of Arteriopathy. Participants African‐American sibships with hypertension in Jackson, Mississippi (N = 1,108). Measurements Waist circumference and body mass index (BMI) were measured at two examinations 5 years apart. Stable adiposity was defined as values within 5% of the first measure. A composite cognitive Z‐score was derived from individual cognitive test Z‐scores at two study visits 6 years apart. Results Larger waist circumference was associated with greater rate of cognitive decline during follow‐up (beta = −0.0009 per year, P = .001); BMI, change in waist circumference, and change in BMI were not associated with rate of decline. Loss of adiposity in midlife was associated with higher cognitive Z‐scores in middle‐aged individuals, and loss of adiposity in late life was associated with lower Z‐scores in older adults (P = .01 for interaction between waist circumference and age; P = .04 for interaction between BMI and age). Simultaneous inclusion of waist circumference and BMI in the cross‐sectional model suggested an association between larger waist circumference and poorer cognitive performance (beta = −0.009, P = .006) and between higher BMI and better cognitive performance (beta = 0.014, P = .06). Conclusion The results suggested a differential pattern of the relationship between adiposity and cognition according to age (mid‐ or late life) and regional distribution of adiposity.