Background Elderly colorectal cancer patients have worse prognosis than younger patients. Age-related survival differences may be cancer or treatment related, but also due to death from other causes. This study aims to compare population-based survival data for young (<65 years), aged (65–74 years), and elderly (≥75 years) colorectal cancer patients. Methods All patients operated for stage I–III colorectal cancer between 1991 and 2005 in the western region of The Netherlands were included. Crude survival, relative survival, and conditional relative survival curves, under the condition of surviving 1 year, were made for colon and rectal cancer patients separately. Furthermore, 30-day, 1-year, and 1-year excess mortality data were compared. Results A total of 9,397 stage I–III colorectal cancer patients were included in this study. Crude survival curves showed clear survival differences between the age groups. These age-related differences were less prominent in relative survival and disappeared in conditional relative survival (CRS). Only in stage III disease did elderly patients have worse CRS than young patients. Furthermore, significant age-related differences in 30-day and 1-year excess mortality were found. Thirty-day mortality vastly underestimated 1-year mortality for all age groups. Conclusions Elderly colorectal cancer patients who survive the first year have the same cancer-related survival as younger patients. Therefore, decreased survival in the elderly is mainly due to differences in early mortality. Treatment of elderly colorectal cancer patients should focus on perioperative care and the first postoperative year.