The US government considers veterans to have been exposed to Agent Orange if they served in Vietnam while the carcinogen was in use, and these veterans are often deemed at high risk of prostate cancer (PCa). Here, we assess whether presumed Agent Orange exposure is independently associated with increased risk of any metastatic or fatal PCa in a diverse Veteran cohort still alive in the modern era (at least 2011), when accounting for race/ethnicity, family history, and genetic risk. Participants in the Million Veteran Program (MVP; enrollment began in 2011) who were on active duty during the Vietnam War era (August 1964-April 1975) were included (n = 301,470). Agent Orange exposure was determined using the US government definition. Genetic risk was assessed via a validated polygenic hazard score. Associations with age at diagnosis of any PCa, metastatic PCa, and death from PCa were assessed via Cox proportional hazards models. On univariable analysis, exposure to Agent Orange was not associated with increased PCa (hazard ratio HR: 1.02, 95% confidence interval CI: 1.00-1.04, p = 0.06), metastatic PCa (HR: 0.98, 95% CI: 0.91-1.05, p = 0.55), or fatal PCa (HR: 0.94, 95% CI: 0.79-1.09, p = 0.41). When accounting for race/ethnicity and family history, Agent Orange exposure was independently associated with slightly increased risk of PCa (HR: 1.06, 95% CI: 1.04-1.09, <10-6) but not with metastatic PCa (HR: 1.07, 95% CI: 0.98-1.15, p = 0.10) or PCa death (HR: 1.02, 95% CI: 0.83-1.23, p = 0.09). Similar results were found when accounting for genetic risk. Agent Orange exposure history may not improve modern PCa risk stratification.