Carotid artery plaque is an established marker of subclinical atherosclerosis with pronounced sex-dimorphism. Here, we aimed to identify genetic variants associated with carotid plaque burden (CPB) and to examine potential sex-specific genetic effects on plaque sizes. We defined six operationalizations of CPB considering plaques in common carotid arteries, carotid bulb, and internal carotid arteries. We performed sex-specific genome-wide association analyses for all traits in the LIFE-Adult cohort (n = 727 men and n = 550 women) and tested significantly associated loci for sex-specific effects. In order to identify causal genes, we analyzed candidate gene expression data for correlation with CPB traits and corresponding sex-specific effects. Further, we tested if previously reported SNP associations with CAD and plaque prevalence are also associated with CBP. We found seven loci with suggestive significance for CPB (p<3.33x10-7), explaining together between 6 and 13% of the CPB variance. Sex-specific analysis showed a genome-wide significant hit for men at 5q31.1 (rs201629990, β = -0.401, p = 5.22x10-9), which was not associated in women (β = -0.127, p = 0.093) with a significant difference in effect size (p = 0.008). Analyses of gene expression data suggested IL5 as the most plausible candidate, as it reflected the same sex-specific association with CPBs (p = 0.037). Known plaque prevalence or CAD loci showed no enrichment in the association with CPB. We showed that CPB is a complementary trait in analyzing genetics of subclinical atherosclerosis. We detected a novel locus for plaque size in men only suggesting a role of IL5. Several estrogen response elements in this locus point towards a functional explanation of the observed sex-specific effect.