Rectal cancer (RC) is a challenging disease to treat that requires chemotherapy, radiation and surgery to optimize outcomes for individual patients. No accurate model of RC exists to answer fundamental research questions relevant to patients. We established a biorepository of 65 patient-derived RC organoid cultures (tumoroids) from patients with primary, metastatic or recurrent disease. RC tumoroids retained molecular features of the tumors from which they were derived, and their ex vivo responses to clinically relevant chemotherapy and radiation treatment correlated with the clinical responses noted in individual patients' tumors. Upon engraftment into murine rectal mucosa, human RC tumoroids gave rise to invasive RC followed by metastasis to lung and liver. Importantly, engrafted tumors displayed the heterogenous sensitivity to chemotherapy observed clinically. Thus, the biology and drug sensitivity of RC clinical isolates can be efficiently interrogated using an organoid-based, ex vivo platform coupled with in vivo endoluminal propagation in animals.