•Evaluation of relations between tool displacements and surface profile formation.•Novel method for measurement of ball end mill’s working part displacements.•Two distinct mechanisms of surface profile formation depended on tool overhang.•High effect of cutting speed on surface roughness in milling with flexible tool. The study presents an analysis of relations between the instantaneous tool displacements and surface roughness formed during ball end milling of surface with inclination towards the tool’s axis. A novel experimental method for the estimation of ball end mill’s working part vibrations, considering displacements correlated with the geometrical errors of tool-toolholder-spindle system and deflections caused by milling forces has been proposed. The experiments have been conducted on hard-to-cut low carbon hardened alloy steel. Milling tests involved the use of monolithic ball end mills varied in terms of overhang l values. In the first stage, the finishing machining of inclined surface with variable cutting speeds vc was carried out. During the experiment, the instantaneous tool’s joining part displacements have been measured with the application of laser displacements sensor. Subsequently, in order to obtain the tool’s tip instantaneous displacements, the analytical extrapolation method has been proposed. In the next stage, the machined surface roughness has been measured with the application of stylus and optical profile meters. The investigations show that the value of tool’s overhang significantly affects the mechanisms of surface roughness generation during finishing ball end milling. In case of milling with the rigid tool (l = 35 mm), the surface roughness is strongly correlated with kinematic-geometric model, as well as with geometrical errors of machining system. Nevertheless, in case of milling with the slender tool (l = 85 mm), the surface roughness formation is mainly affected by the tool’s working part dynamic deflections caused by milling forces.