Since the 2003 heatwave in Europe, evidence has been rapidly increasing on the association between extreme temperature and all-cause mortality. Little is known, however, about cause-specific cardiovascular mortality, effect modification by air pollution and aircraft noise, and which population groups are the most vulnerable to extreme temperature. We conducted a time-stratified case-crossover study in Zurich, Switzerland, including all adult cardiovascular deaths between 2000 and 2015 with precise individual exposure estimates at home location. We estimated the risk of 24,884 cardiovascular deaths associated with heat and cold using distributed non-linear lag models. We investigated potential effect modification of temperature-related mortality by fine particles, nitrogen dioxide, and night-time aircraft noise and performed stratified analyses across individual and social characteristics. We found increased risk of mortality for heat (odds ratio OR = 1.28 95% confidence interval: 1.11–1.49 for 99th percentile of daily Tmean (24 °C) versus optimum temperature at 20 °C) and cold (OR = 1.15 0.95–1.39, 5th percentile of daily Tmean (−3 °C) versus optimum temperature at 20 °C). Heat-related mortality was particularly strong for myocardial infarctions and hypertension related deaths, and among older women (>75 years). Analysis of effect modification also indicated that older women with lower socio-economic position and education are at higher risk for heat-related mortality. PM2.5 increased the risk of heat-related mortality for heart failure, but not all-cause cardiovascular mortality. This study provides useful information for preventing cause-specific cardiovascular temperature-related mortality in moderate climate zones comparable to Switzerland. Display omitted •Temperature related risk of cardiovascular mortality was highest on hot days.•2% of deaths were attributable to hot temperature and 5% to cold temperature.•Older women with lower socio-economic position are more vulnerable to heat.•Housing and air pollution, but not noise, modify vulnerability to temperature.•Understanding effect modifiers is relevant for preventing temperature-related mortality.