We explore star formation histories (SFHs) of galaxies based on the evolution of the star formation rate stellar mass relation (SFR-M). Using data from the FourStar Galaxy Evolution Survey (ZFOURGE) in combination with far-IR imaging from the Spitzer and Herschel observatories we measure the SFR-M relation at 0.5 < z <. Similar to recent works we find that the average infrared spectral energy distributions of galaxies are roughly consistent with a single infrared template across a broad range of redshifts and stellar masses, with evidence for only weak deviations. We find that these two estimates are in broad qualitative agreement, but that there is room for improvement at a more detailed level. At early times the SFHs suggest mass growth rates that are as much as 10 x higher than inferred from the SMF. However, at later times the SFHs under-predict the inferred evolution, as is expected in the case of additional growth due to mergers.