Abstract Background Longitudinal data on the role of atherosclerosis in different vessel beds in the etiology of cognitive impairment and dementia are scarce and inconsistent. Methods Between 2003–2006, 2364 nondemented persons underwent computed tomography of the coronaries, aortic arch, extracranial, and intracranial carotid arteries to quantify atherosclerotic calcification. Participants were followed for incident dementia (n = 90) until April 2012. At baseline and follow-up participants also underwent a cognitive test battery. Results Larger calcification volume in all vessels, except in the coronaries, was associated with a higher risk of dementia. After adjustment for relevant confounders, extracranial carotid artery calcification remained significantly associated with a higher risk of dementia hazard ratio per standard deviation increase in calcification volume: 1.37 (1.05, 1.79). Additional analyses for Alzheimer's disease only or censoring for stroke showed similar results. Larger calcification volumes were also associated with cognitive decline. Conclusions Atherosclerosis, in particular in the extracranial carotid arteries, is related to a higher risk of dementia and cognitive decline.