Context Earlier studies have investigated the role of obesity‐related inflammation and endogenous sex hormones in men. The role of interleukin‐6 (IL‐6) and C‐reactive protein (CRP) with testosterone and sex hormone binding globulin (SHBG) levels in men is still debated. Objective To investigate the independent association between levels of high sensitivity CRP (hsCRP) and IL‐6 with endogenous sex hormones in men. Design Cross‐sectional observational study using data from the Multi‐Ethnic Study of Atherosclerosis. Patients or Other Participants A community‐based sample of 3212 men aged 45–84 years was included. After exclusions, 3041 men remained for the analyses. Main Outcome Measure(s) Serum concentrations of testosterone, SHBG, hsCRP, IL‐6, and sTNFR were measured from the baseline exam. Multivariable linear regressions were used to examine the association of inflammatory markers with sex hormones. Results An inverse association was found between levels of hsCRP and levels of testosterone and SHBG, even after adjustment for confounders and IL‐6 (Total Testosterone; B = −0.14, Bioavailable Testosterone; B = −0.06, and SHBG; B = −0.66). Similar results were found for IL‐6, although a positive association was found for SHBG (B = 0.95). Notably, an inverse association was found for IL‐6 with bioavailable testosterone in African Americans and Hispanic Americans aged 45–54 years. No associations were found for sTNFR and endogenous sex hormones. Conclusion Our results indicate that inflammatory markers have independent associations with levels of testosterone (total and bioavailable) and furthermore, appear to associate differently with SHBG levels.