We present a comprehensive investigation of the cosmological evolution of the luminosity function of galaxies and active galactic nuclei (AGNs) in the infrared (IR). Based on the observed dichotomy in the ages of stellar populations of early-type galaxies on one side and late-type galaxies on the other, the model interprets the epoch-dependent luminosity functions at z > or = 1.5 using a physical approach for the evolution of proto-spheroidal galaxies and of the associated AGNs, while IR galaxies at z < 1.5 are interpreted as being mostly late-type "cold" (normal) and "warm" (star-burst) galaxies. The modeled AGN contributions to the counts and to the cosmic infrared background (CIB) are always sub-dominant. A prediction of the present model, useful to test it, is a systematic variation with wavelength of the populations dominating the counts and the contributions to the CIB intensity. This implies a specific trend for cross-wavelength CIB power spectra, which is found to be in good agreement with the data.