Purpose In this systematic review and meta-analysis, we aimed to estimate cancer-specific mortality and all-cause mortality among cancer survivors associated with both short (typically 5 or 6 h/night) and long (typically 9 or 10 h/night) sleep duration (versus recommendations), separately by sex, cancer site, and sampling frame. Methods We completed a systematic literature search in five databases and captured relevant literature published through December 2018. Two reviewers independently screened 9,823 records and 32 studies were included representing over 73,000 deaths in cancer survivors. Estimates for short and long sleep duration compared to ‘recommended’ were pooled using random-effects models. Results Pooled hazards ratios for short and long sleep duration for all-cancer-specific mortality were 1.03 (95% CI 1.00–1.06) and 1.09 (95% CI 1.04–1.13), respectively. In subgroup analyses by cancer site, statistically significant increased risks were found for both short and long sleep durations for lung cancer-specific mortality. These associations were maintained when stratified by sex and sampling frame. There were no statistically significant associations found between either short or long sleep duration and breast, colorectal, ovarian, or prostate cancer-specific mortality. Statistically significant increases in all-cause mortality were observed with long sleep duration in breast cancer survivors (1.38; 95% CI 1.16–1.64) with no significant associations found for colorectal or liver/pancreatic cancers. Conclusions We observed that long sleep duration increases cancer-specific mortality for all-cancers and lung cancers, while all-cause mortality is increased for breast cancer survivors. Limitations were found within the existing literature that need to be addressed in future studies in order to improve the understanding regarding the exact magnitude of the effect between sleep duration and site-specific mortality.