Adverse effects of breast cancer treatment can negatively affect survivors' work ability. Previous reports lacked detailed clinical data or health-related patient-reported outcomes (PROs) and did not prospectively assess the combined impact of treatment and related sequelae on employment. We used a French prospective clinical cohort of patients with stage I-III breast cancer including 1,874 women who were working and ≥ 5 years younger than legal retirement age (≤ 57 years) at breast cancer diagnosis. Our outcome was nonreturn to work (non-RTW) 2 years after diagnosis. Independent variables included treatment characteristics as well as toxicities (Common Toxicity Criteria Adverse Events CTCAE v4) and PROs (European Organization for Research and Treatment of Cancer EORTC Quality of life Questionnaires, Breast cancer module QLQ-BR23 and Fatigue module QLQ-FA12, Hospital Anxiety and Depression Scale) collected 1 year after diagnosis. Logistic regression models assessed correlates of non-RTW, adjusting for age, stage, comorbidities, and socioeconomic covariates. Two years after diagnosis, 21% of patients had not returned to work. Odds of non-RTW were significantly increased among patients treated with combinations of chemotherapy and trastuzumab (odds ratio OR chemotherapy-hormonotherapy: for chemotherapy-trastuzumab, 2.01; 95% CI, 1.18 to 3.44; for chemotherapy-trastuzumab-hormonotherapy, 1.62; 95% CI, 1.10 to 2.41). Other significant associations with non-RTW included grade ≥ 3 CTCAE toxicities (OR no, 1.59; 95% CI, 1.15 to 2.18), arm morbidity (OR no, 1.59; 95% CI, 1.19 to 2.13), anxiety (OR no, 1.47; 95% CI, 1.02 to 2.11), and depression (OR no, 2.29; 95% CI, 1.34 to 3.91). Receipt of systemic therapy combinations including trastuzumab was associated with increased odds of non-RTW. Likelihood of unemployment was also higher among patients who reported severe physical and psychological symptoms. This comprehensive study identifies potentially vulnerable patients and warrants supportive interventional strategies to facilitate their RTW.