Recently, very robust universal properties have been shown to arise in one-dimensional growth processes with local stochastic rules, leading to the Kardar-Parisi-Zhang (KPZ) universality class. Yet it has remained essentially unknown how fluctuations in these systems correlate at different times. Here, we derive quantitative predictions for the universal form of the two-time aging dynamics of growing interfaces and we show from first principles the breaking of ergodicity that the KPZ time evolution exhibits. We provide corroborating experimental observations on a turbulent liquid crystal system, as well as a numerical simulation of the Eden model, and we demonstrate the universality of our predictions. These results may give insight into memory effects in a broader class of far-from-equilibrium systems.