The shear mechanical behavior of discontinuities under unloading conditions induced by intensive rock mass excavation differs from that under loading conditions. Therefore, the mechanical parameters obtained from the conventional direct shear test cannot effectively be used to assess the stability of excavated rock masses. To solve this problem, we conducted a series of experiments on saw-toothed discontinuities with different undulating angles and initial shear stresses under unloading normal stress with constant shear stress. The test results showed that the shear pattern transformed from climbing to climbing-gnawing and then to gnawing patterns as the undulating angle and initial shear stress increased. The shear stress remained stable with increasing shear displacement after instability in the climbing pattern, decreased with fluctuations in the climbing–gnawing pattern, and dropped steeply in the gnawing pattern. A negative linear correlation exists between the unloading magnitude and the initial shear stress. Unloading normal stress promoted deformation rebound and sawtooth damage, reducing the mobilizing shear strength. Modified Patton and Barton shear strength criteria considering the undulating angle, initial stress, and failure pattern under unloading normal stress were proposed. Highlights The shear mechanical behavior of saw-toothed discontinuities with different undulating angles and initial shear stresses under unloading normal stress with constant shear stress are investigated. The failure pattern changes from the climbing pattern to the climbing–gnawing pattern and then to the gnawing pattern as the undulating angle and the initial shear stress increase. Unloading promotes the increase in rebound deformation and the area of the damage zone, leading to the decrease in shear strength of saw-toothed discontinuities. The modified Patton criterion and the Barton criterion are proposed for unloading normal stress considering the undulating angle, initial stress state, and failure pattern.