Aims. By combining data from the VIMOS VLT Deep Survey (VVDS) with the Spitzer Wide-area InfraRed Extragalactic survey (SWIRE), we have built the currently largest spectroscopic sample of high redshift galaxies selected in the rest-frame near-infrared. We have obtained 2040 spectroscopic redshifts of galaxies with (m_) _{\rm AB} < 21.5 at 3.6 \mu m, and 1255 spectroscopic redshifts of galaxies with (m_) _{\rm AB} < 21. These allow us to investigate the clustering evolution of galaxies selected via their rest- frame near-infrared luminosity in the redshift range 0.2 < z < 2.1. Methods. We use the projected two-point correlation function w_{\rm p}(r_{\rm p}) to study the three dimensional clustering properties of galaxies detected at 3.6 \mu m and 4.5 \mu m with the InfraRed Array Camera (IRAC) in the SWIRE survey with measured spectroscopic redshifts from the first epoch VVDS. We compare these properties to those of a larger sample of 16672 SWIRE galaxies for which we have accurate photometric redshifts in the same field. Results. We find that in the 3.6 \mu m and 4.5 \mu m flux limited samples, the apparent correlation length does not change from redshift similar to 2 to the present. The measured correlation lengths have a mean value of r sub(0) \simeq 3.9\pm0.5 h super(-1) Mpc for the galaxies selected at 3.6 \mu m and a mean value of r sub(0) \simeq 4.4\pm0.5 h super(-1) Mpc for the galaxies selected at 4.5 \mu m, across the whole redshift range explored. These values are larger than those typicaly found for I-band selected galaxies at I_{\rm AB} < 24, for which r sub(0) varies from 2.69 h super(-1) Mpc to 3.63 h super(-1) Mpc between z = 0.5 to z = 2.1. We find that the difference in correlation length between I-band and 3.6-4.5 \mu m selected samples decreases with increasing redshift, becoming comparable at z \simeq 1.5. We interpret this as evidence that galaxies with older stellar populations and galaxies actively forming stars reside in comparably over-dense environments at epochs earlier than z \simeq 1.5, supporting the recently reported flattening of the color-density relation at high redshift. The increasing difference in correlation length with cosmic time observed between rest-frame UV-optical and near-infrared selected samples could then be an indication that star formation is gradually shifting to lower density regions with decreasing redshift, while the older, passively evolving galaxies remain in the most over-dense peaks.