Background To evaluate the quality of life (QOL) of breast cancer survivors who have undergone breast reconstruction and have breast cancer-related lymphedema (BCRL). Methods Patients with a unilateral mastectomy with or without breast reconstruction were evaluated for BCRL and their QOL. Patients were divided into a non-BCRL and a BCRL group. Patients with subjective complaints of arm swelling and/or an interlimb volume difference of >200 mL, or undergoing treatment for arm lymphedema were defined as having BCRL. QOL was assessed using cancer-specific (EORTC QLQ-C30 and EORTC QLQ-B23) and disease specific (Lymph-ICF) questionnaires. Results In total, 253 patients with a mean follow-up time of 51.7 (standard deviation = 18.5) months since mastectomy completed the QOL questionnaires. Of these patients, 116 (46%) underwent mastectomy alone and 137 (54%) had additional breast reconstruction. A comparison of the QOL scores of 180 patients in the non-BCRL group showed a significantly better physical function (p = 0.004) for patients with reconstructive surgery compared with mastectomy patients. In the 73 patients with BCRL, a comparison of the QOL scores showed no significant differences between patients with mastectomy and reconstructive surgery. After adjusting for potential confounders, multivariate analysis showed a significant impact of BCRL on physical function (β =  - 7.46; p = 0.009), role function (β =  - 15.75; p = 0.003), cognitive function (β =  - 11.56; p = 0.005), body vision (β =  - 11.62; p = 0.007), arm symptoms (β = 20.78; p = 0.000), and all domains of the Lymph-ICF questionnaire. Conclusions This study implies that BCRL has a negative effect on the QOL of breast cancer survivors, potentially negating the positive effects on QOL reconstructive breast surgery has.