Purpose This study aimed to investigate changes of QoL during the first 5 years of survival among disease-free Chinese breast cancer survivors. Methods A prospective cohort study enrolled 1462 Chinese women with early-stage breast cancer, and longitudinally visited those patients at four time-points, namely baseline (T0), 18- (T1), 36- (T2), and 60-month (T3) after diagnosis. This study included 992 patients who were disease-free during the first 5 years of survival and who had completed QoL assessments at all four time-points. Results The score of global health status/QoL improved gradually (T1, T2, T3 > T0; P  < 0.001 for overall comparisons). Social functioning score significantly improved when compared to that of T0 (T1, T2, T3 > T0; P  < 0.001 for overall comparisons). In contrast, cognitive functioning score decreased (T0 > T1, T2, T3; P  < 0.001 for overall comparisons). Scores of physical functioning, role functioning and emotional functioning showed a fluctuated picture, with the highest score achieved at T1. In symptoms profile, most of them scored lowest at T1 (best QoL). Multivariate analysis showed that several characteristics significantly correlated to changes in QoL from T0 to T3. For instance, patients with higher education had better recovery of physical functioning, role functioning, and social functioning. Conclusion During the first 5 years of survival, patients’ global health status/QoL improved over time, social functioning consistently improved, but cognitive functioning steadily deteriorated. Most of functioning domains and symptoms improved at 18-month follow-up, but such improvements were not maintained and even deteriorated at 36- and 60-month post-diagnosis. This study suggested that some interventions should be investigated during such period.