We have investigated the stability of the pulse frequency second derivatives (ν) of PSR 0823+26, 1706-16, 1749-28 and 2021+51 that show significant quadratic trends in their pulse-frequency histories in order to determine whether the observed second derivatives are secular or if they arise as part of noise processes. We have used time-of-arrival (TOA) data extending to more than three decades, which are the longest time-spans ever taken into account in pulse-timing analyses. We investigated the stability of the pulse-frequency second derivative in the framework of low-resolution noise power spectra estimated from the residuals of pulse frequency and TOA data. We have found that the ν terms of these sources arise from the red torque noise in the fluctuations of pulse-frequency derivatives, which may originate from the external torques from the magnetosphere of the pulsar.