•The influence of mounting error on tooth profile measurement using a profilometer is confirmed.•A fish-shaped tooth profile form measurement error caused by mounting error has been identified.•A mounting error compensation method based on projection is proposed.•The fish-shaped measurement error can be effectively compensated by the projection method. A profilometer can be used to monitor the tooth surface topography evolution during gear pitting tests. Due to the mounting relationship variation between the instrument and the gear, the tooth profile’s measurement position deviation from its theoretical location will introduce a measurement error. In this study, the tooth profile measurement error introduced by mounting using a tactile profilometer was theoretically analysed and experimentally confirmed as a fish shape. A mounting error compensation method based on projection onto the perpendicular plane to the gear’s datum axis was proposed. The results show that the fish-shaped tooth profile measurement error can be effectively compensated, benefitting the evaluation of the gear tooth surface wear during gear pitting tests.