The aim of the present study was to investigate self-esteem associated factors related to childhood/adolescence or young-adulthood in male and female college students. Based on the i-Share cohort, childhood/adolescence and young-adulthood related factors were collected at study enrolment. Three months later, global self-esteem was evaluated through the Rosenberg Self-esteem Scale. Linear regression models were performed for the different time period factors (childhood/adolescence factors, young-adulthood factors) and R-squared were computed for each model. Among the 6602 college students included, 78.8 % were female with a mean age of 20.7 (SD = 2.2) and mean self-esteem score of 27.6 (SD = 5.7) for females and 30.3 (SD = 5.7) for males. Low self-esteem was associated with factors related to childhood/adolescence (weak parental support, life events, parents with depression or anxiety problems) and young adulthood (freshmen year, high BMI, dissatisfaction with social life, not practicing sports or extra-university activities). Factors related to young adulthood seem to explain more variability among males than factors related to childhood/adolescence (9 % versus 5 %), whereas the opposite was observed for females (5 % versus 7 %). Interventions targeting self-esteem should address several factors that consider period of life and gender to intervene efficiently (during childhood/adolescence for females and young-adulthood for males). •Childhood/adolescence and young adulthood factors explained self-esteem independently from mental health conditions•Childhood/adolescence factors explained more variability in females whereas this were young-adulthood factors in males•Higher effects of parental support and economic situation during childhood on female self-esteem rather than male•Higher effects of sports practice and social life during young-adulthood on male self-esteem rather than female•Gender differences regarding period of life should guide interventional research to promote college students’ self-esteem