ABSTRACT We study the temporal evolution of downflows observed at the lateral edges of penumbral filaments in a sunspot located very close to the disk center. Our analysis is based on a sequence of nearly diffraction-limited scans of the Fe i 617.3 nm line taken with the CRisp Imaging Spectro-Polarimeter instrument at the Swedish 1 m Solar Telescope. We compute Dopplergrams from the observed intensity profiles using line bisectors and filter the resulting velocity maps for subsonic oscillations. Lateral downflows appear everywhere in the center-side penumbra as small, weak patches of redshifts next to or along the edges of blueshifted flow channels. These patches have an intermittent life and undergo mergings and fragmentations quite frequently. The lateral downflows move together with the hosting filaments and react to their shape variations, very much resembling the evolution of granular convection in the quiet Sun. There is a good relation between brightness and velocity in the center-side penumbra, with downflows being darker than upflows on average, which is again reminiscent of quiet Sun convection. These results point to the existence of overturning convection in sunspot penumbrae, with elongated cells forming filaments where the flow is upward but very inclined, and weak lateral downward flows. In general, the circular polarization profiles emerging from the lateral downflows do not show sign reversals, although sometimes we detect three-lobed profiles that are suggestive of opposite magnetic polarities in the pixel.