Abstract Background Physical activity may be beneficial for cognition, but the effect may vary depending on personal characteristics. Methods We investigated the associations between leisure-time physical activity (LTPA) from mid- to late life, the risk of dementia, and the role of body mass index, sex, and APOE in the CAIDE study during 28-year follow-up. Cognitive function of a random subsample was assessed at a mean age of 78.8 years (n = 1511), and dementia/Alzheimer's disease (AD) diagnoses were identified from national registers for the entire target population (n = 3559). Results Moderate (hazard ratio HR, 1.46; 95% confidence interval CI, 1.08–1.99) and low levels of midlife LTPA (HR, 1.39; 95% CI, 0.99–1.95) were associated with higher risk of dementia in comparison with the most active category. The benefits were more pronounced among men, overweight individuals, and APOE ε4 noncarriers. Maintaining high LTPA (HR, 0.16; 95% CI, 0.06–0.41) or increasing LTPA (HR, 0.19; 95% CI, 0.09–0.40) after midlife was associated with lower dementia risk. Similar results were observed for AD. Conclusions The window of opportunity for preventive physical activity interventions may extend from midlife to older ages.