There is evidence that birth size is positively associated with height in later life, but it remains unclear whether this is explained by genetic factors or the intrauterine environment. To analyze the associations of birth weight, length and ponderal index with height from infancy through adulthood within mono- and dizygotic twin pairs, which provides insights into the role of genetic and environmental individual-specific factors. This study is based on the data from 28 twin cohorts in 17 countries. The pooled data included 41,852 complete twin pairs (55% monozygotic and 45% same-sex dizygotic) with information on birth weight and a total of 112,409 paired height measurements at ages ranging from 1 to 69 years. Birth length was available for 19,881 complete twin pairs, with a total of 72,692 paired height measurements. The association between birth size and later height was analyzed at both the individual and within-pair level by linear regression analyses. Within twin pairs, regression coefficients showed that a 1-kg increase in birth weight and a 1-cm increase in birth length were associated with 1.14–4.25 cm and 0.18–0.90 cm taller height, respectively. The magnitude of the associations was generally greater within dizygotic than within monozygotic twin pairs, and this difference between zygosities was more pronounced for birth length. Both genetic and individual-specific environmental factors play a role in the association between birth size and later height from infancy to adulthood, with a larger role for genetics in the association with birth length than with birth weight. •Birth weight and length are positively associated with later height from infancy to adulthood.•Both genetic and individual-specific environmental factors influence the association between birth size and later height.•Genetic factors are more importantly involved in the association with birth length than with birth weight.