Physical frailty has been recognized as a clinical syndrome resulting from declines in various physiological systems; however, the role of the central nervous system in the pathophysiology of frailty remains unclear. The I‐Lan Longitudinal Aging Study randomly sampled community‐dwelling people aged 50 or older for a brain magnetic resonance imaging study. All participants were assessed for frailty status (robust, prefrail, and frail) based on the presence of five frailty components: slow walking speed, muscle weakness, low physical activity, exhaustion and weight loss (Fried criteria). Gray matter volume (GMV) changes associated with frailty status and individual frailty components were examined. Overall, 456 participants (64.0 ± 8.5 years, 47.6% women) were included in this study. The prefrail (n = 178, 39.0%) and frail (n = 19, 4.2%) subjects were grouped for analysis. The prefrail–frail group showed reduced GMV, compared to the robust group (n = 259, 56.8%), in the cerebellum, hippocampi, middle frontal gyri, and several other cerebral regions (corrected P < 0.05). Each frailty component was associated with GMV changes in functionally related brain areas. Hierarchical cluster analysis categorized these components into three subsets. Motor‐related components, including weakness, low activity, and slowness, comprised one subset with a common cerebellar involvement. Exhaustion and weight loss were the other two subsets without cerebellar changes. To conclude, physical frailty is associated with a decreased reserve in specific brain regions, especially cerebellum. Further longitudinal studies are needed to explore if the cerebellum‐ and noncerebellum‐based frailty components reflect a distinctive future risk for developing frailty. Hum Brain Mapp 36:3666–3676, 2015. © 2015 Wiley Periodicals, Inc.