Physical activity has been positively associated with gray‐matter integrity. In contrast, pro‐inflammatory cytokines seem to have negative effects on the aging brain and have been related to dementia. It was investigated whether an inactive lifestyle and high levels of inflammation resulted in smaller gray‐matter volumes and predicted cognitive decline across 6 years in a population‐based study of older adults (n = 414). Self‐reported physical activity (fitness‐enhancing, health‐enhancing, inadequate) was linked to gray‐matter volume, such that individuals with inadequate physical activity had the least gray matter. There were no overall associations between different pro‐and anti‐inflammatory markers (IL‐1β, IL‐6, IL‐10, IL‐12p40, IL‐12p70, G‐CSF, and TNF‐α) and gray‐matter integrity. However, persons with inadequate activity and high levels of the pro‐inflammatory marker IL‐12p40 had smaller volumes of lateral prefrontal cortex and hippocampus and declined more on the Mini‐Mental State Examination test over 6 years compared with physically inactive individuals with low levels of IL‐12p40 and to more physically active persons, irrespective of their levels of IL‐12p40. These patterns of data suggested that inflammation was particularly detrimental in inactive older adults and may exacerbate the negative effects of physical inactivity on brain and cognition in old age. Hum Brain Mapp 37:3462–3473, 2016. © 2016 Wiley Periodicals, Inc.