A new dynamic model of a variable-length rope, which could be used for the transient analysis of a buoy-rope-generator (BRG) wave energy system, was proposed in this paper. The model started from the basic dynamic equations of variable mass system, and took into account the physical properties such as axial force, shear and bending. According to the principle of D’Alembert-Lagrange, the equivalent integral weak formulation was firstly obtained, and through consistent linearization and isoparametric discretization, the finite element model of the variable-length rope was then derived. The Bathe scheme was employed to solve the model numerically, based on its excellent performance in solving nonlinear dynamic problems, and an automatic time step size algorithm was designed according to the number of iterations of the two substeps of Bathe scheme. The procedures of rope mesh regeneration were also put forward, where only one variable-length element was always located at the top end of the rope, and the rest were all fixed-length elements. The proposed variable-length rope model and solution schemes were verified through comparison with the results of a tank experiment. Finally, the transient dynamics of a kind of BRG system was analyzed and discussed.