Abstract Cross-sectional neuroimaging studies suggest that hippocampal and prefrontal cortex functions underlie individual differences in memory ability in older individuals, but it is unclear how individual differences in cognitive ability in youth contribute to cognitive and neuroimaging measures in older age. Here, we investigated the relative influences of midlife memory ability and age-related memory change on memory-related BOLD-signal variability at one time point, using a sample from a longitudinal population-based aging study (N = 203, aged 55–80 years). Hierarchical regression analyses showed that midlife memory ability, assessed 15–20 years earlier, explained at least as much variance as memory change in clusters in the left inferior prefrontal cortex and the bilateral hippocampus, during memory encoding. Furthermore, memory change estimates demonstrated higher sensitivity than current memory levels in identifying distinct frontal regions where activity was selectively related to age-related memory change, as opposed to midlife memory. These findings highlight challenges in interpreting individual differences in neurocognitive measures as age-related changes in the absence of longitudinal data and also demonstrate the improved sensitivity of longitudinal measures.