Astrophys.J.624:L81-L84,2005 We present deep 3.6 - 8 micron imaging of the Hubble Deep Field South with IRAC on the Spitzer Space Telescope. We study Distant Red Galaxies (DRGs) at z>2 selected by Js - Ks > 2.3 and compare them to a sample of Lyman Break Galaxies (LBGs) at z=2-3. The observed UV-to-8 micron spectral energy distributions are fit with stellar population models to constrain star formation histories and derive stellar masses. We find that 70% of the DRGs are best described by dust-reddened star forming models and 30% are very well fit with old and ``dead'' models. Using only the I - Ks and Ks - 4.5 micron colors we can effectively separate the two groups. The dead systems are among the most massive at z~2.5 (mean stellar mass <M*> = 0.8 x 10^11 Msun) and likely formed most of their stellar mass at z>5. To a limit of 0.5 x 10^11 Msun their number density is ~10 x lower than that of local early-type galaxies. Furthermore, we use the IRAC photometry to derive rest-frame near-infrared J, H, and K fluxes. The DRGs and LBGs together show a large variation (a factor of 6) in the rest-frame K-band mass-to-light ratios (M/L_K), implying that even a Spitzer 8 micron-selected sample would be very different from a mass-selected sample. The average M/L_K of the DRGs is about three times higher than that of the LBGs, and DRGs dominate the high-mass end. The M/L_K ratios and ages of the two samples appear to correlate with derived stellar mass, with the most massive galaxies being the oldest and having the highest mass-to-light ratios, similar as found in the low-redshift universe.