Abstract Magnetic systems governed by exchange interactions between magnetic moments harbor frustration that leads to ground state degeneracy and results in the new topological state often referred to as a frustrated state of matter (FSM). The frustration in the commonly discussed magnetic systems has a spatial origin. Here we demonstrate that an array of nanomagnets coupled by the real retarded exchange interactions develops a new state of matter, time frustrated matter (TFM). In a spin system with the time-dependent retarded exchange interaction, a single spin-flip influences other spins not instantly but after some delay. This implies that the sign of the exchange interaction changes, leading to either ferro- or antiferromagnetic interaction, depends on time. As a result, the system’s temporal evolution is essentially non-Markovian. The emerging competition between different magnetic orders leads to a new kind of time-core frustration. To establish this paradigmatic shift, we focus on the exemplary system, a granular multiferroic, where the exchange transferring medium has a pronounced frequency dispersion and hence develops the TFM.