Introduction In animal models, refined carbohydrates (RF) worsen Alzheimer's disease (AD). However, the long‐term effects of high RF intake on the risk of dementia and AD are poorly described in epidemiological studies. Moreover, the interaction between RF and the apolipoprotein E ε4 allele (APOE‐ε4) is unknown. Our study investigated whether RF‐rich diets are associated with the risk of dementia and AD. Methods The glycemic load (GL) was quantified in 2777 elderly participants from the French Three‐City Study to estimate RF intake. Then, the associations between GL and risk of dementia and AD, and the interaction with APOE‐ε4 over a 12‐year period were assessed using proportional hazards models. Results After adjustment for potential confounders, high afternoon‐snack GL was associated with increased dementia and AD risk in APOE‐ε4 carriers (hazard ratio = 1.27 1.03–1.56). Discussion This study highlights that RF‐rich diets are a risk factor for dementia and AD in APOE‐ε4 carriers.