We investigated the association of load and accumulation of white matter hyperintensities (WMHs) with rate of cognitive decline. This population-based study included 510 dementia-free people (age ≥60 years) who had repeated measures of global and regional (lobar, deep, periventricular) WMHs up to 6 years (from 2001–2003 to 2007–2010) and repeated measures of cognitive function (episodic memory, semantic memory, category fluency, letter fluency, executive function, perceptual speed) up to 15 years (from 2001–2004 to 2016–2019). We found that greater baseline loads of global and regional WMHs were associated with faster decline in letter fluency, perceptual speed, and global cognition. Furthermore, faster accumulation of global, deep, and periventricular WMHs was related to accelerated cognitive decline, primarily in perceptual speed. These data show that WMHs are associated with decline in perceptual speed rather than episodic or semantic memory and that cognitive change is more vulnerable to WMH accumulations in deep and periventricular regions.