Although high T-cell density is a well-established favorable prognostic factor in colorectal cancer, the prognostic significance of tumor-associated plasma cells, neutrophils, and eosinophils is less well-defined. We computationally processed digital images of hematoxylin and eosin (H&E)-stained sections to identify lymphocytes, plasma cells, neutrophils, and eosinophils in tumor intraepithelial and stromal areas of 934 colorectal cancers in two prospective cohort studies. Multivariable Cox proportional hazards regression was used to compute mortality HR according to cell density quartiles. The spatial patterns of immune cell infiltration were studied using the G function, which estimates the likelihood of any tumor cell in a sample having at least one neighboring immune cell of the specified type within a certain radius. Validation studies were performed on an independent cohort of 570 colorectal cancers. Immune cell densities measured by the automated classifier demonstrated high correlation with densities both from manual counts and those obtained from an independently trained automated classifier (Spearman's ρ 0.71-0.96). High densities of stromal lymphocytes and eosinophils were associated with better cancer-specific survival < 0.001; multivariable HR (4th vs 1st quartile of eosinophils), 0.49; 95% confidence interval, 0.34-0.71. High G area under the curve (AUC ; = 0.002) and high G AUC ( < 0.001) also showed associations with better cancer-specific survival. High stromal eosinophil density was also associated with better cancer-specific survival in the validation cohort ( < 0.001). These findings highlight the potential for machine learning assessment of H&E-stained sections to provide robust, quantitative tumor-immune biomarkers for precision medicine.