Time-varying mesh stiffness, caused by the change of tooth contact number and contact position, is one of the main sources of vibration of a gear transmission system. In order to comprehensively understand the vibration properties of a planetary gear set, it is necessary to evaluate the mesh stiffness effectively. When a crack happens in one gear, the mesh stiffness will decrease and consequently the vibration properties of the gear system will change. This change of vibration can be characterized through dynamic simulation of a gearbox and processed further to detect the crack severity and location. In this paper, the potential energy method is used to analytically evaluate the mesh stiffness of a planetary gear set. A modified cantilever beam model is used to represent the external gear tooth and derive the analytical equations of the bending, shear and axial compressive stiffness. A crack propagation model is developed and the mesh stiffness reduction is quantified when a crack occurs in the sun gear or the planet gear. •A beam model is developed for the perfect external gear tooth.•Advantages of the proposed beam model are emphasized.•A crack propagation model is proposed for the external gear with crack.•Mesh stiffness equations are derived analytically for the external gear.•Mesh stiffness reduction is quantified when a crack occurs.