Due to significant galaxy contamination and impurity in stellar mass selected samples (up to 95 per cent from z = 0-3), we examine the star formation history, quenching time-scales, and structural evolution of galaxies using a constant number density selection with data from the United Kingdom Infra-Red Deep Sky Survey Ultra-Deep Survey field. Using this methodology, we investigate the evolution of galaxies at a variety of number densities from z = 0-3. We find that samples chosen at number densities ranging from 3 x 10 super( -4) to 10 super( -5) galaxies Mpc-3 (corresponding to z ~ 0.5 stellar masses of M* = 10 super( 10.95-11.6) M0) have a star-forming blue fraction of ~50 per cent at z ~ 2.5, which evolves to a nearly 100 per cent quenched red and dead population by z ~ 1. We also see evidence for number density downsizing, such that the galaxies selected at the lowest densities (highest masses) become a homogeneous red population before those at higher number densities. Examining the evolution of the colours for these systems furthermore shows that the formation redshift of galaxies selected at these number densities is zform > 3. The structural evolution through size and Sersic index fits reveal that while there remains evolution in terms of galaxies becoming larger and more concentrated in stellar mass at lower redshifts, the magnitude of the change is significantly smaller than for a mass-selected sample. We also find that changes in size and structure continues at z < 1, and is coupled strongly to passivity evolution. We conclude that galaxy structure is driving the quenching of galaxies, such that galaxies become concentrated before they become passive.