This paper proposes a new iterative compensation methodology of geometric errors to improve the machining accuracy of a non-orthogonal five-axis machine tool (NOFAMT). Firstly, based on homogeneous transform matrix (HTM) and multi-body system (MBS) theory, the relative motion constraint equations (TRMCEs) of the tool tip position and tool orientation vector related to a NOFAMT with a nutating rotary B axis are established. Then, by utilizing TRMCEs, the mapping relationships between tool path and the numerical control (NC) command without and with considering the geometric errors are constructed respectively. In order to truly reproduce tool motion trajectory of the machine tool driven by the given NC command, the mapping relationship between the NC command and tool cutting trajectory is also established. Meanwhile, procedures of iterative compensation are described by using the aforementioned mapping relationships without the traditional inverse calculation, and the actual NC code is generated in self-developed compensation software. It is not difficult to find that the new approach takes the difference between tool path and tool cutting trajectory as the control objective and can directly obtain the actual NC code controlling the machine tool to achieve the desired machining accuracy. Finally, a cutting test is carried out on the DMU60P NOFAMT. Experimental results show the developed iterative compensation methodology is precise and effective for NOFAMTs. Therefore, compared with the existing methods, the new method is more direct and accurate. And its basic idea can be applied to other type of machine tools. •First, the relative motion constraint equations of a non-orthogonal five-axis machine tool are established.•Second, the mutual mapping relationships between tool path, NC command and tool cutting trajectory are established respectively.•Third, a new iterative compensation methodology is presented.•Fourth, a cutting experiment is proposed to verify the machining accuracy improvement of the researched machine tool.