Colorectal cancer is the third most commonly diagnosed cancer in Australia. Emerging evidence from several countries suggests increasing incidence in people aged <50 years. We assessed colon and rectal cancer incidence trends in people aged 20+ in Australia from 1982 to 2014. We used data on 375,008 incident cases (248,162 colon and 126,846 rectal). We quantified the annual percentage change (APC) in rates by age group using Joinpoint regression. For people aged <50 years, colon cancer rates increased from the mid-2000s, with the increase in APCs ranging from 1.7% to 9.3% per annum (depending on specific age group); rectal cancer rates increased from the early 1990s, with APCs ranging from 0.9% to 7.1% per annum. For people aged 50 to 69 years, colon and rectal cancer rates decreased from the mid-1990s, with the decrease in APCs in specific age groups ranging from 0.8% to 4.8% per annum (except for colon cancer in those ages 65 to 69 years, where similar rate decreases were observed from 2007). An overall reduction in older persons (>70 years) was estimated at 1.9% to 4.9% per annum for colon cancer from 2010 onward and 1.1% to 1.8% per annum in rectal cancer from the early 2000s onward. Colon and rectal cancer incidence has increased in people aged <50 years in Australia over the last two decades. However, colon and rectal cancer rates decreased in people aged 50+, likely due to and organized bowel cancer screening. Further research is needed to examine the cause of the increase and to quantify the impact of future trends on the cost-effectiveness of population-based screening for those <50 years.