Purpose Clinical trials showing programmed death (PD)-1–PD-ligand 1 (L1) axis as a promising therapeutic target in melanoma and non-small cell lung cancers have made the pathway a focus of recent attention. However, the data regarding PD-L1/PD-1 in breast cancer are inconsistent. Given the heterogeneity of breast cancers, the clinical relevance of PD-L1 and PD-1 tumor infiltrating lymphocytes (TIL) may vary according to subtypes of breast cancer. We aim to investigate PD-L1 expression in a large cohort of breast cancers and analyze its clinico-pathological as well as outcome relationship according to molecular subtypes. Also, we evaluate the relationship of PD-1 TIL and PD-L1 expression with patients’ survival, particularly for breast cancers with high TIL. Methods and results Immunohistochemical analysis of PD-L1 on tissue arrays for 1091 breast cancer patients and PD-1 TIL on 97 whole sections was performed. Associations of PD-L1 with luminal cancers ( p  < 0.001) and features associated with that subtype lower histologic grade, absence of necrosis, ER, PR, and AR expression ( p  < 0.001) were observed. However, in HER2+ breast cancers, PD-L1 was an independent poor prognostic indicator (DFS: HR = 1.866, p  = 0.001; OS: HR = 1.517, p  = 0.036). Interestingly, HER2+ cancers showed a lower PD-1 TIL level compared to the other high TIL cases ( p  = 0.011). Cases with low PD-TIL but high PD-L1 expression showed the worst survival. This could be indicative of an active immune suppression by PD-L1 expression. Conclusions Our data showed the relevance of PD-L1 expression in HER2+ breast cancer. A combined evaluation of PD-L1 and PD-1 TIL in the prognosis of breast cancer might also be of value in treatment prediction.