The cumulative emission resulting from hadronic cosmic-ray interactions in star-forming galaxies (SFGs) has been proposed as the dominant contribution to the astrophysical neutrino flux at TeV to PeV energies reported by IceCube. The same particle interactions also inevitably create γ-ray emission that could be detectable as a component of the extragalactic γ-ray background (EGB), which is now measured with the Fermi-LAT in the energy range from 0.1 to 820 GeV. New studies of the blazar flux distribution at γ-ray energies above 50 GeV place an upper bound on the residual non-blazar component of the EGB. We show that these results are in strong tension with models that consider SFGs as the dominant source of the diffuse neutrino backgrounds. A characteristic spectral index for parent cosmic rays in starburst galaxies of ΓSB 2.3 for dN / dE ∝ E − Γ SB is consistent with the observed scaling relation between γ-ray and IR luminosity for SFGs, the bounds from the non-blazar EGB, and the observed γ-ray spectra of individual starbursts, but underpredicts the IceCube data by approximately an order of magnitude.