This paper provides a novel approach for identifying the process damping directly from chatter-free milling tests. First, power spectrum density matrix of cutter's deflections is theoretically derived by introducing transfer function and random excitation force, and the spectral decomposition of the power spectrum density matrix is formulated as an explicit function of modal parameters. Then, exponential attenuation method is adopted to extract the damping ratios from the inverse Fourier transformation result of the decomposed form. Finally, tangential and radial ploughing force coefficients, which are utilized to characterize process damping, are simultaneously calculated based on energy balance principle. Besides, experimental setup consisting of displacement sensors is specially designed to measure the cutter's deflections, which are further used to calculate the power spectrum density required in the above identification procedure. It is experimentally proven that the accuracy of chatter stability limits in milling process is improved when the proposed process damping model is considered. •Process damping is directly identified from chatter-free milling tests based on operational modal analysis.•Power spectrum density matrix of cutter's deflections is analytically derived as an explicit function of modal parameters.•Experimental setup suitable for measuring the cutter's deflections is designed.•Effectiveness of the proposed process damping model is verified by chatter experiments.