Purpose High-density tumor-infiltrating lymphocytes (TILs) are a prognostic marker for triple-negative breast cancer (TNBC). However, lymphocytic infiltration is heterogeneous in its pattern. We aimed to explore the utility of TIL distribution patterns against TIL density for predicting TNBC prognosis and chemotherapeutic effects. Methods Primary invasive TNBC cases were retrieved from a single institutional cohort, and archived samples were reviewed by two board-certificated pathologists. We used 154 consecutive surgical specimens from patients with standard adjuvant therapy, and 80 biopsies taken before primary systemic chemotherapy. The average density of stromal TILs was scored at 10% intervals, while the distribution pattern of TILs was evaluated as diffuse or non-diffuse. The association between TILs and prognosis or pathological complete response (pCR) was statistically analyzed. Results A diffuse pattern of TILs at primary surgery correlated with better prognosis (relapse-free survival RFS, hazard ratio HR 3.71, 95% confidence interval CI 1.60–8.57; overall survival OS, HR 3.87, 95% CI 1.46–10.27), as well as high TIL density (≥ 50%; RFS, HR 4.51, 95% CI 2.06–9.90; OS, HR 3.28, 95% CI 1.32–8.14). Diffuse TIL pattern and nodal status were independent prognostic factors in multivariate analysis. Diffuse TIL pattern upon biopsy was associated with higher pCR rate (diffuse, 46%; non-diffuse, 21%; P  = 0.032). All high TIL cases had diffuse patterns and the best outcome. Interobserver concordance was moderate ( k  = 0.53–0.55; distribution pattern) to good (weighted k  = 0.67–0.69; density), and it was faster to assess the distribution pattern than to assess the density of TIL. Conclusions Showing similar clinical impacts to the TIL density, diffuse TILs could be a predictive marker for better prognosis and higher pCR. The assessment of TIL distribution pattern is simple, faster, and practical. Heterogeneous tumor immunity may contribute to further stratification of TNBC treatment.