At high elevation, temperate trees generally exhibit adaptive genetic differentiation in their morphological and physiological traits. On account of this directional selection, we hypothesized that tree populations growing near their upper cold elevational limits exhibit lower phenotypic plasticity of growth and leaf morphological traits in response to temperature changes than populations growing at lower elevations. Seedlings of six common deciduous tree species originating from low and high elevations were transplanted into eight common gardens along two elevational gradients in the Swiss Alps. The aim of the experiment was to evaluate the genetic differentiation in growth and leaf morphology between populations from low and high elevations and to quantify the phenotypic plasticity of these traits to temperature changes. In contrast to growth that decreased with increasing elevation, leaf mass per area (LMA) showed no significant change with elevation of common garden, except for a decrease in Laburnum alpinum for both low‐ and high‐elevation provenances. Interestingly, leaf density was found to decrease with elevation of the gardens for all species. Genetic differentiation between low‐ and high‐elevation populations was found in both leaf morphology and growth: high‐elevation populations tended to have slower growth rate than low‐elevation populations, while no consistent trend was found for LMA across species. Interestingly, for Acer pseudoplatanus and Fraxinus excelsior high‐elevation populations exhibited a lower phenotypic plasticity of growth in response to temperature compared to lower populations, whereas no interactions between the elevation of a provenance and the elevation of the garden was detected for the four other species. Hence, during young life stages, the expected increase in tree growth in future warmer climates might be lower in populations living in the coldest part of the species distribution range in temperate species such as Acer pseudoplatanus and Fraxinus excelsior, but similar in other tree species, disregarding other environmental changes.