In machining process, cutting forces are difficult to measure accurately due to the dynamic characteristic of measurement system. The cutting forces compensation requires the frequency response function (FRF), which cannot be measured in actual production. This paper proposes an effective method for FRF modelling and calculating based on transfer path analysis (TPA) and receptance coupling substructure analysis (RCSA). The measurement system is divided into three substructures: workpiece, table dynamometer with screws, joint surface between two previous substructures. The joint surface is simplified as a spring-damping model with contact stiffness and damping. The FRF of workpiece is obtained by finite element method (FEM), which is adjustable for different workpieces and tool positions. The FRF of dynamometer can be measured only by single impact test. The contact parameters can be identified by a few impact tests, which can be applied to other cutting conditions. The FRF between tool tip and dynamometer output is first derived based on TPA and RCSA. Second, a joint surface parameters identification algorithm as well as its simplified algorithm is presented through a few impact tests. Finally, impact tests and milling tests are implemented for parameters identification, algorithm verification and cutting forces compensation. The experimental results show that the proposed method has sufficient accuracy for the assembly FRF prediction. •The RFR from tool tip to dynamometer is derived based on TPA and RCSA.•The joint surface parameters identification algorithm is presented based on a few impact tests.•The simplified algorithm for parameters identification is drived.•A number of tests are implemented and validate the effectiveness of the proposed method.