In the high speed milling, the tool run-out affects the cutting force greatly and results in pre-matured tool life. To investigate this relationship, an improved instantaneous milling force per tooth is proposed, with inclusion of tool run-out effect. The un-deformed chip thickness considering tool run-out are defined and modeled, according to the geometrical relationships and axial milling ranges per tooth. Meanwhile, instead of the studying the conventional average flank wear, tool wear per tooth is studied for more sensitive correlation with force. Based on milling tests with Inconel 718, the error of the force model prediction is found less than 1% against the experimental data, and the correlation between the axial instantaneous milling force and tool wear per tooth is above 0.9. The results have shown that the proposed model can accurately describe the instantaneous force per tooth including tool run out effect, and the axial force component is a good indication of tool wear condition. •Modeled the instantaneous force per tooth in milling, with run-out effect considerations.•Determined explicitly the upper and lower boundaries of the cutter-workpiece engagement.•Validated the model's accuracy with experimental data.•Identified that the z-axial instantaneous force sensitive to tool wear per tooth.