Purpose The aim of this study was to evaluate the impact of time to radiotherapy (TTR) after completion of chemotherapy (CT), and TTR after surgery, in breast cancer (BC) patients. Patients and Methods Continuous breast cancer patients treated with surgery and CT followed by radiotherapy (RT) from 2009 through 2015 were retrospectively reviewed. Patients were categorized into four groups with respect to TTR after CT, i.e. <4, 4–8, 8–12, and >12 weeks, and TTR after surgery, i.e. <147, 147–180, 180–202, and >202 days. The Cox proportional hazards model was used to identify the independent effect of TTRs. Results Overall, 989 patients were enrolled. Patients with a TTR of >12 weeks after CT showed significantly worse breast cancer-specific survival (BCSS) and overall survival (OS) compared with those who had a TTR of <4 weeks (BCSS: hazard ratio HR 0.28, 95% confidence interval CI 0.1–0.76; OS: HR 0.33, 95% CI 0.13–0.88), 4–8 weeks (BCSS: HR 0.23, 95% CI 0.08–0.66; OS: HR 0.29, 95% CI 0.11–0.8), and 8–12 weeks (BCSS: HR 0.22, 95% CI 0.05–0.96; OS: HR 0.23, 95% CI 0.06–0.99). TTR after surgery showed no significant association with survival outcomes in the entire cohort, except in patients with hormone receptor (HR)-positive disease and those receiving mastectomy. In HR-positive tumors, a TTR after CT of >12 weeks remained an independent predictor for adverse BCSS and OS. Conclusion Initiation of RT beyond 12 weeks after CT might compromise survival outcomes. Efforts should be made to avoid delaying RT, especially after completion of CT and in patients with HR-positive tumors, positive lymph nodes, and those receiving mastectomy.