To determine whether the tumor immune infiltrate, as recently evaluated with the Immunoscore methodology, could be a useful prognostic marker in patients with rectal cancers. The influence of the immune infiltrate on patient's outcome was investigated in patients with or without preoperative chemoradiation therapy (pCRT). The density of total (CD3(+)) and cytotoxic (CD8(+)) T lymphocytes was evaluated by immunohistochemistry and quantified by a dedicated image analysis software in surgical specimens of patients with rectal cancer (n = 111) who did not receive pCRT and in tumor biopsies performed before pCRT from additional 55 patients. The results were correlated with tumor recurrence, patient's survival, and response to pCRT. The densities of CD3(+) and CD8(+) lymphocytes and the associated Immunoscore (from I0 to I4) were significantly correlated with differences in disease-free and overall survival (HR, 1.81 and 1.72, respectively; all P < 0.005). Cox multivariate analysis supports the advantage of the Immunoscore compared with the tumor-node-metastasis (TNM) staging in predicting recurrence and survival (all P < 0.001). Lymph node ratio added information in a prognostic model (all P < 0.05). In addition, high infiltration of CD3(+) and CD8(+) lymphocytes in tumor biopsies was associated with downstaging of the tumor after pCRT (CD3(+) cells; Fisher exact test P = 0.01). The Immunoscore could be a useful prognostic marker in patients with rectal cancer treated by primary surgery. The determination of the immune infiltrate in biopsies before treatment could be a valuable information for the prediction of response to pCRT.