•In mammals, specific periods of the life span can be regarded as sensitive phases.•We focus on the prenatal, the early postnatal and the adolescent phases.•Adversity across phases often reduces affiliative social behavior in later life.•This may adjust the individual’s social phenotype to ‘tough times ahead’.•Underlying mechanisms can involve hormones, epigenetics and the microbiome. Here, we summarize recent advances on how environmental influences during sensitive phases alter the social behavioral phenotype of rodents later in life. Current studies support the view that the prenatal, early postnatal and adolescent periods of life can be regarded as sensitive phases. Environmental cues acting on the organism during these phases have a wide variety of effects on adult social behavior. One pattern that emerges across species and sensitive phases is that adversity tends to reduce social interactions and particularly affiliative social behavior. Concerning underlying mechanisms, various hormones can be involved; however, glucocorticoids frequently serve as the signal instigating plasticity. There is also increasing appreciation of non-endocrine mechanisms, specifically epigenetics and the microbiome. Concerning function, some evidence exists that sensitive phase outcomes adjust the individual’s social phenotype to the nature of the social environment to be present during adulthood and breeding, though additional empirical support is still needed.