Helical gears are generally considered to be more stable than spur gears. But rattling of the helical gear transmission is found in the engineering practice. The torsional dynamics equations of helical gear pair in high-speed railway gearbox are established in order to reveal the rattling mechanism of helical gear transmission. Double and three teeth pair drive-side meshing are considered. The multi-state meshing zone, load distribution rate and time-varying stiffness determined by contact ratio are analyzed and calculated. The dynamic characteristic transition process of the system is analyzed according to the bifurcation diagrams and the corresponding top Lyapunov exponent (TLE) diagrams, phase portraits, Poincaré maps and time history spectrums of dynamic meshing force based on the calculation of these parameters. The tooth disengagement, tooth back-side contact and their parameter range are found. This study can provide theoretical basis for rattling suppression and transmission stability improvement of helical gear pair.