Triaxial loading and unloading tests on marble specimens under different stress paths were conducted to investigate the characteristics of energy evolution in rock deformation process. Results show that tensile failure occurred in rock specimens under uniaxial compression, while shear failure dominated under triaxial loading and unloading. The energy storage limit of rock specimens under triaxial loading was higher than that under uniaxial compression or triaxial unloading. A nonlinear energy evolution model of rock was established based on the interaction mechanism of energy accumulation and energy dissipation. Results from the theoretical model are in good agreement with the test results. Since the evolution of energy was characterised by bifurcation and chaos, a strain value corresponding to the energy iterative growth factor at the first bifurcation of the energy equation was chosen as the initiation criterion for rock failure. The critical strain accounted for 77%, 72–76%, and 72–81% of the peak strain under uniaxial compression, triaxial loading, and unloading, respectively.