Understanding the climatic drivers of present-day agricultural yields is critical for prioritizing adaptation strategies to climate change. However, unpacking the contribution of different environmental stressors remains elusive in large-scale observational settings in part because of the lack of an extensive long-term network of soil moisture measurements and the common seasonal concurrence of droughts and heat waves. In this study, we link state-of-the-art land surface model data and fine-scale weather information with a long panel of county-level yields for six major US crops (1981-2017) to unpack their historical and future climatic drivers. To this end, we develop a statistical approach that flexibly characterizes the distinct intra-seasonal yield sensitivities to high-frequency fluctuations of soil moisture and temperature. In contrast with previous statistical evidence, we directly elicit an important role of water stress in explaining historical yields. However, our models project the direct effect of temperature-which we interpret as heat stress-remains the primary climatic driver of future yields under climate change.