Purpose Breast cancer patients aged 65+ (“older”) vary in frailty status. We tested whether a deficits accumulation frailty index predicted long-term mortality. Methods Older patients ( n  = 1280) with non-metastatic, invasive breast cancer were recruited from 78 Alliance sites from 2004 to 2011, with follow-up to 2015. Frailty categories (robust, pre-frail, and frail) were based on 35 baseline illness and function items. Cox proportional hazards and competing risk models were used to calculate all-cause and breast cancer-specific mortality for up to 7 years, respectively. Potential covariates included demographic, psychosocial, and clinical factors, diagnosis year, and care setting. Results Patients were 65–91 years old. Most (76.6%) were robust; 18.3% were pre-frail, and 5.1% frail. Robust patients tended to receive more chemotherapy ± hormonal therapy (vs. hormonal) than pre-frail or frail patients (45% vs. 37 and 36%, p  = 0.06), and had the highest adherence to hormonal therapy. The adjusted hazard ratios for all-cause mortality ( n  = 209 deaths) were 1.7 (95% CI 1.2–2.4) and 2.4 (95% CI 1.5–4.0) for pre-frail and frail versus robust women, respectively, with an absolute mortality difference of 23.5%. The adjusted hazard of breast cancer death ( n −99) was 3.1 (95% CI 1.6–5.8) times higher for frail versus robust patients (absolute difference of 14%). Treatment differences did not account for the relationships between frailty and mortality. Conclusions Most older breast cancer patients are robust and could consider chemotherapy where otherwise indicated. Patients who are frail or pre-frail have elevated long-term all-cause and breast cancer mortality. Frailty indices could be useful for treatment decision-making and care planning with older patients.