We quantify the fraction of the cosmic infrared background (CIB) that originates from galaxies identified in the UV/optical/near-infrared by stacking 81,250 (~35.7 arcmin super(-2)) K-selected sources (K sub(AB) < 24.0) split according to their rest-frame U - V versus V - J colors into 72,216 star-forming and 9034 quiescent galaxies, on maps from Spitzer/MIPS (24 mu m), Herschel/PACS (100, 160 mu m), Herschel/SPIRE (250, 350, 500 mu m), and AzTEC (1100 mu m). Of that total, about 95% originates from star-forming galaxies, while the remaining 5% is from apparently quiescent galaxies. The CIB at lambda lap 200 mu m appears to be sourced predominantly from galaxies at z lap 1, while at lambda gap 200 mu the bulk originates from 1 lap z lap 2. Stacking analyses were performed using simstack, a novel algorithm designed to account for possible biases in the stacked flux density due to clustering. It is made available to the public at www.astro.caltech.edu/~viero/viero_homepage/toolbox.html.