Abstract We set out to quantify the number density of quiescent massive compact galaxies at intermediate redshifts. We determine structural parameters based on i-band imaging using the Canada–France–Hawaii Telescope (CFHT) equatorial Sloan Digital Sky Survey (SDSS) Stripe 82 (CS82) survey (∼170 deg2) taking advantage of an exquisite median seeing of ∼0.6 arcsec. We select compact massive (M ⋆ > 5 × 1010 M⊙) galaxies within the redshift range of 0.2 < z < 0.6. The large volume sampled allows to decrease the effect of cosmic variance that has hampered the calculation of the number density for this enigmatic population in many previous studies. We undertake an exhaustive analysis in an effort to untangle the various findings inherent to the diverse definition of compactness present in the literature. We find that the absolute number of compact galaxies is very dependent on the adopted definition and can change up to a factor of >10. We systematically measure a factor of ∼5 more compacts at the same redshift than what was previously reported on smaller fields with Hubble Space Telescope (HST) imaging, which are more affected by cosmic variance. This means that the decrease in number density from z ∼ 1.5 to z ∼ 0.2 might be only of a factor of ∼2–5, significantly smaller than what was previously reported. This supports progenitor bias as the main contributor to the size evolution. This milder decrease is roughly compatible with the predictions from recent numerical simulations. Only the most extreme compact galaxies, with R eff < 1.5 × (M ⋆/1011 M⊙)0.75 and M ⋆ > 1010.7 M⊙, appear to drop in number by a factor of ∼20 and hence likely experience a noticeable size evolution.