Immune infiltration of breast tumours is associated with clinical outcome. However, past work has not accounted for the diversity of functionally distinct cell types that make up the immune response. The aim of this study was to determine whether differences in the cellular composition of the immune infiltrate in breast tumours influence survival and treatment response, and whether these effects differ by molecular subtype. We applied an established computational approach (CIBERSORT) to bulk gene expression profiles of almost 11,000 tumours to infer the proportions of 22 subsets of immune cells. We investigated associations between each cell type and survival and response to chemotherapy, modelling cellular proportions as quartiles. We found that tumours with little or no immune infiltration were associated with different survival patterns according to oestrogen receptor (ER) status. In ER-negative disease, tumours lacking immune infiltration were associated with the poorest prognosis, whereas in ER-positive disease, they were associated with intermediate prognosis. Of the cell subsets investigated, T regulatory cells and M0 and M2 macrophages emerged as the most strongly associated with poor outcome, regardless of ER status. Among ER-negative tumours, CD8+ T cells (hazard ratio HR = 0.89, 95% CI 0.80-0.98; p = 0.02) and activated memory T cells (HR 0.88, 95% CI 0.80-0.97; p = 0.01) were associated with favourable outcome. T follicular helper cells (odds ratio OR = 1.34, 95% CI 1.14-1.57; p < 0.001) and memory B cells (OR = 1.18, 95% CI 1.0-1.39; p = 0.04) were associated with pathological complete response to neoadjuvant chemotherapy in ER-negative disease, suggesting a role for humoral immunity in mediating response to cytotoxic therapy. Unsupervised clustering analysis using immune cell proportions revealed eight subgroups of tumours, largely defined by the balance between M0, M1, and M2 macrophages, with distinct survival patterns by ER status and associations with patient age at diagnosis. The main limitations of this study are the use of diverse platforms for measuring gene expression, including some not previously used with CIBERSORT, and the combined analysis of different forms of follow-up across studies. Large differences in the cellular composition of the immune infiltrate in breast tumours appear to exist, and these differences are likely to be important determinants of both prognosis and response to treatment. In particular, macrophages emerge as a possible target for novel therapies. Detailed analysis of the cellular immune response in tumours has the potential to enhance clinical prediction and to identify candidates for immunotherapy.