The aim of this paper is to investigate the brittle failure mechanism of rock slopes with non-persistent en-echelon joints using the extended finite element method (XFEM). In the research, a novel geometrical method distinguished from the conventional level set method is proposed to describe the cracks in the XFEM. The numerical form of the J-integral considering the effect of the bode force and the frictional contact force is provided. Moreover, a refined coalescence method is proposed. Then, the state function employed to define the state of the crack and the method used to define the crack growth increment are introduced. Next, some numerical tests are carried out to validate the feasibility of the XFEM. Finally, two typical rock slope modes are provided to illustrate the mechanism of the step-path brittle failure of rock slopes. •A novel geometric method is proposed to describe the cracks in the framework of XFEM.•The numerical form of the J-integral considering the effect of bode force and the frictional contact is derived.•A refined coalescence method is proposed and an empirical method to define the crack growth increment is introduced.•Two typical rock slope modes are provided to illustrate the mechanism of the step-path brittle failure of rock slopes.