Self-reported leisure-time physical activity (PA) has previously been linked to risk of cardiovascular disease (CVD). We now aim to investigate the strength of associations between PA and different CVDs and how the risk varies with age. PA and traditional CV risk factors assessed by a questionnaire on a four-level scale in 2,175 men at age 50 years in the ULSAM study. Examinations were thereafter repeated at ages 60, 70, and 77. During 40 years follow-up, 883 individuals experienced a CVD (myocardial infarction, stroke, or heart failure). Using data from all four examinations, a graded reduction in risk of incident CVD was seen with increasing PA (HR 0.84, 95%CI; 0.77-0.93, p = 0.001 for trend test). PA was related to myocardial infarction (HR 0.84, 95%CI; 0.74-0.95, 490 cases), heart failure (HR 0.79, 95%CI; 0.68-0.91, 356 cases), but only of borderline significance vs ischemic stroke (HR 0.85, 95%CI; 0.73-1.00, 315 cases) when the CVDs were analyzed separately. Adjusting for traditional CV risk factors attenuated all relationships between PA and incident CVD, and PA did not improve discrimination of CVD when added on top of risk factors. When 10-year risk was calculated from each examination, age 70 was the time-point when PA was most closely related to incident CVD. Leisure-time physical activity is related to future CVD. This was most evident at 70 years of age. If a causal relationship between self-reported PA and CVD exists, this relationship might to a major degree be mediated by traditional risk factors.