We use a large suite of hydrodynamical simulations of binary galaxy mergers to construct and calibrate a physical prescription for computing the effective radii and velocity dispersions of spheroids. We implement this prescription within a semi-analytic model embedded in merger trees extracted from the Bolshoi Λ cold dark matter N-body simulation, accounting for spheroid growth via major and minor mergers and disc instabilities. We find that without disc instabilities, our model does not predict sufficient numbers of intermediate-mass early-type galaxies in the local Universe. Spheroids also form earlier in models with spheroid growth via disc instabilities. Our model correctly predicts the normalization, slope, and scatter of the low-redshift size–mass and Fundamental Plane relations for early-type galaxies. It predicts a degree of curvature in the Faber–Jackson relation that is not seen in local observations, but this could be alleviated if higher mass spheroids have more bottom-heavy initial mass functions. The model also correctly predicts the observed strong evolution of the size–mass relation for spheroids out to higher redshifts, as well as the slower evolution in the normalization of the Faber–Jackson relation. We emphasize that these are genuine predictions of the model since it was tuned to match hydrodynamical simulations and not these observations.