We construct rest-frame luminosity functions (LFs) at 3.6, 4.5, 5.8, 8 and 24 μm over the redshift range 0 < z < 2 for galaxies and 0 < z < 4 for optical quasi-stellar objects (QSOs), using optical and infrared (IR) data from the Spitzer Wide-area Infrared Extragalactic (SWIRE) Survey. The 3.6- and 4.5-μm galaxy LFs show evidence for moderate positive luminosity evolution up to z∼ 1.5, consistent with the passive ageing of evolved stellar populations. Their comoving luminosity density was found to evolve passively, gradually increasing out to z∼ 0.5-1 but flattening, or even declining, at higher redshift. Conversely, the 24-μm galaxy LF, which is more sensitive to obscured star formation and/or active galactic nuclei (AGN) activity, undergoes strong positive evolution, with the derived IR energy density and star formation rate (SFR) density ∝ (1 +z)γ with γ= 4.5+0.7 −0.6 and the majority of this evolution occurring since z∼ 1. Optical QSOs, however, show positive luminosity evolution in all bands, out to the highest redshifts (3 < z < 4). Modelling as L*∝ (1 +z)γ gave γ= 1.3+0.1 −0.1 at 3.6 μm, γ= 1.0+0.1 −0.1 at 4.5 μm and stronger evolution at the longer wavelengths (5.8, 8 and 24 μm), of γ∼ 3. Comparison of the galaxy LFs to predictions from a semi-analytic model based on cold dark matter (CDM) indicates that an initial mass function (IMF) skewed towards higher mass star formation in bursts compared to locally be preferred. As a result, the currently inferred massive SFRs in distant submm sources may require substantial downwards revision.