The cumulative emission resulting from hadronic cosmic-ray interactions in star-forming galaxies (SFGs) has been proposed as the dominant contribution to the astrophysical neutrino flux at TeV to PeV energies reported by IceCube. The same particle interactions also inevitably create \(\gamma\)-ray emission that could be detectable as a component of the extragalactic \(\gamma\)-ray background (EGB), which is now measured with the Fermi-LAT in the energy range from 0.1 to 820 GeV. New studies of the blazar flux distribution at \(\gamma\)-ray energies above 50 GeV place an upper bound on the residual non-blazar component of the EGB. We show that these results are in strong tension with models that consider SFGs as the dominant source of the diffuse neutrino backgrounds. A characteristic spectral index for parent cosmic rays in starburst galaxies of \(\Gamma_{\rm SB} \simeq 2.3\) for \(dN/dE \propto E^{-\Gamma_{\rm SB}}\) is consistent with the observed scaling relation between \(\gamma\)-ray and IR luminosity for SFGs, the bounds from the non-blazar EGB, and the observed \(\gamma\)-ray spectra of individual starbursts, but underpredicts the IceCube data by approximately an order of magnitude.