Summary It has been increasingly recognized at the basic science level that perturbations in ceramide metabolism are associated with the development and progression of many age‐related diseases. However, the translation of this work to the clinic has lagged behind. Understanding the factors longitudinally associated with plasma ceramides and dihydroceramides (DHCer) at the population level and how these lipid levels change with age, and by sex, is important for the clinical development of future therapeutics and biomarkers focused on ceramide metabolism. We, therefore, examined factors cross‐sectionally and longitudinally associated with plasma concentrations of ceramides and DHCer among Baltimore Longitudinal Study of Aging participants (n = 992; 3960 total samples), aged 55 years and older, with plasma at a mean of 4.1 visits (range 2–6). Quantitative analyses were performed on a high‐performance liquid chromatography‐coupled electrospray ionization tandem mass spectrometer. Linear mixed models were used to assess the relationships between plasma ceramide and DHCer species and demographics, diseases, medications, and lifestyle factors. Women had higher plasma concentrations of most ceramide and DHCer species and showed steeper trajectories of age‐related increases compared to men. Ceramides and DHCer were more associated with waist–hip ratio than body mass index. Plasma cholesterol and triglycerides, prediabetes, and diabetes were associated with ceramides and DHCer, but the relationship showed specificity to the acyl chain length and saturation. These results demonstrate the importance of examining the individual species of ceramides and DHCer, and of establishing whether intra‐individual age‐ and sex‐specific changes occur in synchrony to disease onset and progression.