Germ-line mutations in the exonuclease domains of the POLE and POLD1 genes are associated with an increased, but yet unquantified, risk of colorectal cancer (CRC). We identified families with POLE or POLD1 variants by searching PubMed for relevant studies prior to October 2016 and by genotyping 669 population-based CRC cases diagnosed in patients under 60 years of age, from the Australasian Colorectal Cancer Family Registry. We estimated the age-specific cumulative risks (penetrance) using a modified segregation analysis. We observed 67 CRCs (mean age at diagnosis = 50.2 (SD = 13.8) years) among 364 first- and second-degree relatives from 41 POLE families, and 6 CRCs (mean age at diagnosis = 39.7 (SD = 6.83) years) among 69 relatives from 9 POLD1 families. We estimated risks of CRC up to the age of 70 years (95% confidence interval) for males and females, respectively, to be 28% (95% CI, 10–42%) and 21% (95% CI, 7–33%) for POLE mutation carriers and 90% (95% CI, 33–99%) and 82% (95% CI, 26–99%) for POLD1 mutation carriers. CRC risks for POLE mutation carriers are sufficiently high to warrant consideration of colonoscopy screening and implementation of management guidelines recommended for MSH6 mutation carriers in cases of Lynch syndrome. Refinement of estimates of CRC risk for POLD1 carriers is needed; however, clinical management recommendations could follow those made for POLE carriers.