Background The prognostic significance of tumor-infiltrating lymphocytes (TILs) has been established in breast cancers with estrogen receptor (ER)-negative and human epithelial growth factor receptor 2 (HER2)-negative or HER2-positive subtypes; however, its utility concerning the ER + /HER2 − subtype remains unclear. Methods We evaluated the prognostic value of TILs by analyzing 717 invasive breast cancer operation cases. TILs were classified into three groups based on the proportion of area within the tumor: low ( < 10%), intermediate (10–50%), and high ( > 50%). Disease-free survival (DFS) and overall survival (OS) were calculated according to TIL levels. Results Although there was no significant association between TIL levels and DFS or OS in all patients, high TILs were significantly associated with favorable DFS in Ki67-high ( n  = 238, p  = 0.035) but not in Ki67-low ( n  = 470, p  = 0.46) breast cancers. Multivariable analysis showed that high TILs were a significant and independent factor for DFS (HR 0.34; 95% CI 0.10–0.87; p  = 0.023) among the Ki67-high group. In the ER + /HER2 − subtype, high-TILs showed favorable DFS in the Ki67-high group, although this was not statistically significant ( p  = 0.48); in contrast, unfavorable DFS was observed in the Ki67-low group ( p  = 0.027). Conclusions In Ki67-high breast cancers, high TILs were associated with favorable DFS, irrespective of subtype, but increasing TIL levels correlated with worse DFS in the Ki67-low group with the ER + /HER2 − subtype. These results highlight variation in TIL prognostic significance between Ki67-high and -low breast cancers, particularly for the ER + /HER2 − subtype.