In this paper, we focus on the numerical solution of the second kind of Volterra integral equation with a highly oscillatory Fourier kernel. Based on the calculation of the modified moments, we propose four collocation methods to solve the equations: direct linear interpolation, direct higher order interpolation, direct Hermite interpolation and piecewise Hermite interpolation. These four methods are simple to construct and can quickly compute highly oscillatory integrals involving Fourier functions. We present the corresponding error analysis and it is easy to see that, in some cases, our proposed method has a fast convergence rate in solving such equations. In some cases, our proposed methods have significant advantages over the existing methods. Some numerical experiments demonstrating the efficiency of the four methods are also presented.