•A rigid-flexible coupling model for planetary gear systems is developed.•An approach for simulating the pass effect is proposed.•Validation is performed through ANSYS modal analysis and vibration experiment.•Pre-emphasized noise excitation is introduced, aligning well with experiment. A rigid-flexible coupling model for the planetary gear system is developed, in which the flexibility of the ring gear, input shaft, and carrier are considered. Subsequently, rotational modal projection is proposed to simulate the meshing of the flexible ring gear at any angle of planet gear revolution, and the coordinate transformation equations are presented to establish the coupling between rigid body and rotating flexible body. These two approaches incorporate the pass effect into the coupling model. Additionally, a method employing pre-emphasized noise to simulate random excitations is introduced. The model is validated by comparing computed modal frequencies and dynamic responses with those from ANSYS software and experiment, respectively. Results indicate that the proposed method accurately simulates the modulation signal observed in the experiment. Closely matched resonances excited by random excitations are apparent in experimental and simulated acceleration spectra, highlighting the validity of the proposed method and aiding in the determination of operational system modes. In addition, analyses of parameter influences are conducted, which further reveal the dynamic characteristics of the system.