Vibrations of gear transmission systems (GTSs) are the key problems in the machinery industry. In the previous vibration analysis of GTS, the rotor is usually regarded as a rigid part. The rigid-shaft gear-transmission dynamic system RGDS) model cannot solve the differences of vibrations of rotor at different axial positions. To solve this problem, a flexible-shaft gear- transmission system dynamic (FGDS) model is established by using the finite element method. In the FGDS, the GTSs is separated into some flexible shaft segments, gear meshing unit and bearing unit. The bearing stiffness, bearing contact force, time-varying meshing stiffness and damping force of gear are considered. The time- and frequency-domain vibrations of FGDS and RGDS model under different rotational speeds are compared and analyzed. Note that the FGDS can obtain more accurate results than the RGDS model. The vibrations from FGDS are less than those from RGDS model. This paper can give some new approach for the dynamic modeling and vibration analysis of flexible GTSs.