Prostate cancer risk is influenced by rare and common germline variants. We examined the aggregate association of rare germline pathogenic/likely pathogenic/deleterious (P/LP/D) variants in ATM, BRCA2, PALB2, and NBN with a polygenic risk score (PRS) on prostate cancer risk among 1,796 prostate cancer cases (222 metastatic) and 1,424 controls of African ancestry. Relative to P/LP/D non-carriers at average genetic risk (33%-66% of PRS), men with low (0%-33%) and high (66%-100%) PRS had Odds Ratios (ORs) for overall prostate cancer of 2.08 95% confidence interval (CI) = 0.58-7.49 and 18.06 (95% CI = 4.24-76.84) among P/LP/D carriers and 0.57 (95% CI = 0.46-0.71) and 3.02 (95% CI = 2.53-3.60) among non-carriers, respectively. The OR for metastatic prostate cancer was 2.73 (95% CI = 0.24-30.54) and 28.99 (95% CI = 4.39-191.43) among P/LP/D carriers and 0.54 (95% CI = 0.31-0.95) and 3.22 (95% CI = 2.20-4.73) among non-carriers, for men with low and high PRS, respectively. Lifetime absolute risks of overall prostate cancer increased with PRS (low to high) from 9.8% to 51.5% in P/LP/D carriers and 5.5% to 23.9% in non-carriers. Lifetime absolute risks of metastatic prostate cancer increased with PRS from 1.9% to 18.1% in P/LP/D carriers and 0.3% to 2.2% in non-carriers These findings suggest that assessment of prostate cancer risk for rare variant carriers should include PRS status. These findings highlight the importance of considering rare and common variants to comprehensively assess prostate cancer risk in men of African ancestry.