Regorafenib is synergistic with immune checkpoint inhibition in colorectal cancer preclinical models. This was a single-arm, multicentric phase II trial. Regorafenib was given 3 weeks on/1 week off, 160 mg every day; avelumab 10 mg/kg i.v. was given every 2 weeks, beginning at cycle 1, day 15 until progression or unacceptable toxicity. The primary endpoint was the confirmed objective response rate under treatment, as per RECIST 1.1. The secondary endpoints included a 1-year nonprogression rate, progression-free survival (PFS), and overall survival (OS), safety and biomarkers studies performed on sequential tumor samples obtained at baseline and at cycle 2 day 1. Forty-eight patients were enrolled in four centers. Forty-three were assessable for efficacy after central radiological review. Best response was stable disease for 23 patients (53.5%) and progressive disease for 17 patients (39.5%). The median PFS and OS were 3.6 months 95% confidence interval (CI), 1.8-5.4 and 10.8 months (95% CI, 5.9-NA), respectively. The most common grade 3 or 4 adverse events were palmar-plantar erythrodysesthesia syndrome ( = 14, 30%), hypertension ( = 11, 23%), and diarrhea ( = 6, 13%). High baseline infiltration by tumor-associated macrophages was significantly associated with adverse PFS (1.8 vs. 3.7 months; = 0.002) and OS (3.7 months vs. not reached; = 0.002). Increased tumor infiltration by CD8 T cells at cycle 2, day 1 as compared with baseline was significantly associated with better outcome. The combination of regorafenib + avelumab mobilizes antitumor immunity in a subset of patients with microsatellite stable colorectal cancer. Computational pathology through quantification of immune cell infiltration may improve patient selection for further studies investigating this approach.