The Balloon-borne Large Aperture Submillimeter Telescope (BLAST) has made 1 deg2, deep, confusion-limited maps at three different bands, centered on the Great Observatories Origins Deep Survey South Field. By calculating the covariance of these maps with catalogs of 24 Delta *mm sources from the Far-Infrared Deep Extragalactic Legacy Survey, we have determined that the total submillimeter intensities are 8.60 +/- 0.59, 4.93 +/- 0.34, and 2.27 +/- 0.20 nW m-2 sr-1 at 250, 350, and 500 Delta *mm, respectively. These numbers are more precise than previous estimates of the cosmic infrared background (CIB) and are consistent with 24 Delta *mm-selected galaxies generating the full intensity of the CIB. We find that the fraction of the CIB that originates from sources at z >= 1.2 increases with wavelength, with 60% from high-redshift sources at 500 Delta *mm. At all BLAST wavelengths, the relative intensity of high-z sources is higher for 24 Delta *mm-faint sources than that for 24 Delta *mm-bright sources. Galaxies identified as active galactic nuclei (AGNs) by their Infrared Array Camera colors are 1.6-2.6 times brighter than the average population at 250-500 Delta *mm, consistent with what is found for X-ray-selected AGNs. BzK-selected galaxies are found to be moderately brighter than typical 24 Delta *mm-selected galaxies in the BLAST bands. These data provide high-precision constraints for models of the evolution of the number density and intensity of star-forming galaxies at high redshift.