After resection, colorectal cancer liver metastases (CRLM) surrounded by a desmoplastic rim carry a better prognosis than the metastases replacing the adjacent liver. However, these histopathological growth patterns (HGPs) are insufficient to guide clinical decision-making. We explored whether the adaptive immune features of HGPs could refine prognostication. From 276 metastases resected in 176 patients classified by HGPs, tissue microarrays were used to assess intratumoral T cells (CD3), antigen presentation capacity (MHC class I) and CD73 expression producing immunosuppressive adenosine. We tested correlations between these variables and patient outcomes. The 101 (57.4%) patients with dominant desmoplastic HGP had a median recurrence-free survival (RFS) of 17.1 months compared to 13.3 months in the 75 patients (42.6%) with dominant replacement HGP (p = 0.037). In desmoplastic CRLM, high vs. low CD73 was the only prognostically informative immune parameter and was associated with a median RFS of 12.3 months compared to 26.3, respectively (p = 0.010). Only in dominant replacement CRLM, we found a subgroup (n = 23) with high intratumoral MHC-I expression but poor CD3 T cell infiltration, a phenotype associated with a short median RFS of 7.9 months. Combining the assessments of HGP and adaptive immune features in resected CRLM could help identify patients at risk of early recurrence.