Colorectal cancer (CRC) shows variable response to immune checkpoint blockade, which can only partially be explained by high tumor mutational burden (TMB). We conducted an integrated study of the cancer tissue and associated tumor microenvironment (TME) from patients treated with pembrolizumab (KEYNOTE 177 clinical trial) or nivolumab to dissect the cellular and molecular determinants of response to anti- programmed cell death 1 (PD1) immunotherapy. We selected multiple regions per tumor showing variable T-cell infiltration for a total of 738 regions from 29 patients, divided into discovery and validation cohorts. We performed multiregional whole-exome and RNA sequencing of the tumor cells and integrated these with T-cell receptor sequencing, high-dimensional imaging mass cytometry, detection of programmed death-ligand 1 (PDL1) interaction in situ, multiplexed immunofluorescence, and computational spatial analysis of the TME. In hypermutated CRCs, response to anti-PD1 immunotherapy was not associated with TMB but with high clonality of immunogenic mutations, clonally expanded T cells, low activation of Wnt signaling, deregulation of the interferon gamma pathway, and active immune escape mechanisms. Responsive hypermutated CRCs were also rich in cytotoxic and proliferating PD1+CD8 T cells interacting with PDL1+ antigen-presenting macrophages. Our study clarified the limits of TMB as a predictor of response of CRC to anti-PD1 immunotherapy. It identified a population of antigen-presenting macrophages interacting with CD8 T cells that consistently segregate with response. We therefore concluded that anti-PD1 agents release the PD1-PDL1 interaction between CD8 T cells and macrophages to promote cytotoxic antitumor activity. Display omitted Colorectal cancers responsive to anti-programmed cell death 1 immunotherapy show clonal immunogenic mutations, low Wnt activation, beta-2-microglobulin deregulation, and high infiltration of antigen presenting macrophages interacting with programmed cell death 1-positive cluster of differentiation 8 T cells.