There is a differential in prostate cancer mortality between black and white men. Advances in precision medicine have shifted the research focus toward underlying genetic differences. However, nonbiological factors may have a large role in these observed disparities. Therefore, we sought to measure the relative importance of race compared to health care and social factors on prostate cancer specific mortality. Using the SEER (Surveillance, Epidemiology, and End Results) database we identified 514,878 men diagnosed with prostate cancer at age 40 years or greater between 2004 and 2012. We also selected a subset of black and white men matched by age, stage and birth year. We stratified patients by age 40 to 54, 55 to 69 and 70 years or older and disease stage, resulting in 18 groups. By applying random forest methods with variable importance measures we analyzed 15 variables and interactions across 4 categories of factors (tumor characteristics, race, and health care and social factors) and the relative importance for prostate cancer specific mortality. Tumor characteristics at diagnosis were the most important factors for prostate cancer mortality. Across all groups race was less than 5% as important as tumor characteristics and only more important than health care and social factors in 2 of the 18 groups. Although race had a significant impact, health care and social factors known to be associated with racial disparities had greater or similarly important effects across all ages and stages. Eradicating disparities in prostate cancer survival will require a multipronged approach, including advances in precision medicine. Disparities will persist unless health care access and social equality are achieved among all populations.