Summary This 3-year longitudinal study among older adults showed that declining muscle mass, strength, power, and physical performance are independent contributing factors to increased fear of falling, while declines of muscle mass and physical performance contribute to deterioration of quality of life. Our findings reinforce the importance of preserving muscle health with advancing age. Introduction The age-associated loss of skeletal muscle quantity and function are critical determinants of independent physical functioning in later life. Longitudinal studies investigating how decrements in muscle components of sarcopenia impact fear of falling (FoF) and quality of life (QoL) in older adults are lacking. Methods Twenty-six healthy older subjects (age, 74.1 ± 3.7; Short Physical Performance Battery (SPPB) score ≥10) and 22 mobility-limited older subjects (age, 77.2 ± 4.4; SPPB score ≤9) underwent evaluations of lower extremity muscle size and composition by computed tomography, strength and power, and physical performance at baseline and after 3-year follow-up. The Falls Efficacy Scale (FES) and Short Form-36 questionnaire (SF-36) were also administered at both timepoints to assess FoF and QoL, respectively. Results At 3-year follow-up, muscle cross-sectional area (CSA) ( p  < 0.013) and power decreased ( p  < 0.001), while intermuscular fat infiltration increased ( p  < 0.001). These decrements were accompanied with a longer time to complete 400 m by 22 ± 46 s ( p  < 0.002). Using linear mixed-effects regression models, declines of muscle CSA, strength and power, and SPPB score were associated with increased FES score ( p  < 0.05 for each model). Reduced physical component summary score of SF-36 over follow-up was independently associated with decreased SPPB score ( p  < 0.020), muscle CSA ( p  < 0.046), and increased 400 m walk time ( p  < 0.003). Conclusions In older adults with and without mobility limitations, declining muscle mass, strength, power, and physical performance contribute independently to increase FoF, while declines of muscle mass and physical performance contribute to deterioration of QoL. These findings provide further rationale for developing interventions to improve aging muscle health.