Mismatch repair (MMR) deficiency is an indicator of good prognosis in localized colon cancer but also associated with lack of expression of caudal-type homeobox transcription factor 2 (CDX2) and high tumor grade; markers that in isolation indicate a poor prognosis. Our study aims to identify clinically relevant prognostic subgroups by combining information about tumor grade, MMR phenotype, and CDX2 expression. Immunohistochemistry for MMR proteins and CDX2 was performed in 544 patients with colon cancer stage II–III, including a cohort from a randomized trial. In patients with proficient MMR (pMMR) and CDX2 negativity, hazard ratio (HR) for cancer death was 2.93 (95% CI 1.23–6.99, p = 0.015). Cancer-specific survival for pMMR/CDX2-negative cases was 35.8 months (95% CI 23.4–48.3) versus 52.1–53.5 months (95% CI 45.6–58.6, p = 0.001) for the remaining cases (CDX2-positive tumors or deficient MMR (dMMR)/CDX2-negative tumors). In our randomized cohort, high tumor grade was predictive of response to adjuvant fluorouracil–levamisole in pMMR patients, with a significant interaction between tumor grade and treatment (p = 0.036). For pMMR patients, high tumor grade was a significant marker of poor prognosis in the surgery-only group (HR 4.60 (95% CI 1.68–12.61), p = 0.003) but not in the group receiving chemotherapy (HR 0.66 (95% CI 0.15–3.00), p = 0.587). To conclude, patients with pMMR and CDX2 negativity have a very poor prognosis. Patients with pMMR and high-graded tumors have a poor prognosis but respond well to adjuvant chemotherapy. CDX2 expression and tumor grade did not impact prognosis in patients with dMMR.