Large-scale genomic characterization of tumors from prospective cohort studies may yield new insights into cancer pathogenesis. We performed whole-exome sequencing of 619 incident colorectal cancers (CRCs) and integrated the results with tumor immunity, pathology, and survival data. We identified recurrently mutated genes in CRC, such as BCL9L, RBM10, CTCF, and KLF5, that were not previously appreciated in this disease. Furthermore, we investigated the genomic correlates of immune-cell infiltration and found that higher neoantigen load was positively associated with overall lymphocytic infiltration, tumor-infiltrating lymphocytes (TILs), memory T cells, and CRC-specific survival. The association with TILs was evident even within microsatellite-stable tumors. We also found positive selection of mutations in HLA genes and other components of the antigen-processing machinery in TIL-rich tumors. These results may inform immunotherapeutic approaches in CRC. More generally, this study demonstrates a framework for future integrative molecular epidemiology research in colorectal and other malignancies. Display omitted •Whole-exome sequencing of 619 colorectal cancers with clinicopathologic annotations•Discovery of significantly mutated genes in colorectal cancer•Neoantigen load correlation with infiltrating lymphocytes and memory T cells•Positive selection for HLA mutations in immune-cell-infiltrated tumors Through whole-exome sequencing of annotated colorectal tumors, Giannakis et al. identify additional colorectal cancer driver genes and correlate high neoantigen load with increased lymphocytic infiltration and improved survival. They also find positive selection for HLA mutations in immune-cell-infiltrated tumors. These results may inform immunotherapeutic approaches in colorectal cancer.