In this paper, a numerical model based on the coupling algorithm of SPH-FEM is established and verified by laboratory experiments, the fragmentation mechanism of granite impacted by the abrasive water jet is discussed. Compared with the experimental results, the error of erosion depth is less than 5.46%, the error of influence range around the punching is less than 3.89%, and the error of punching diameter is less than 7.16%. The numerical model shows that the material removal is usually caused by the diffusion around the jet stream, and the total amount is 84.06%. The results show that under the strong impact of the jet, it will penetrate all substances in the jet path. When the kinetic energy of the jet is decreases, the removal of materials usually follows the weak path, that is, through the cleavage plane of the feldspar grains and the contact interface between the mineral crystals. Display omitted •A numerical model of heterogeneous granite is established.•The influence of granite minerals on the removal mechanism of AWJ is discussed.•The crushing process of granite impacted by AWJ is reproduced by simulation.•Two kinds of material removal forms of granite impacted by AWJ are described.