•An overall gear transmission model which incorporates with gearbox casing is proposed.•Introduced a new way to model gear tooth pitting and spalling.•The proposed model in simulating tooth pitting and spalling is validated by experimental tests.•The vibration fault features of tooth pitting and spalling are discussed. Dynamical modeling of a gear system with faults has been an important research topic for understanding fault features and their associated fault vibration mechanisms. Due to the complicated structures and intricate interactions between the components of the gear system, the fault vibration features and corresponding vibration mechanisms due to tooth pitting and spalling remain mostly unknown. This paper proposes a novel spur gear dynamical model, validated by various experimental tests, to analytically investigate the effects of tooth pitting and spalling on the vibration responses of a gear transmission. The proposed dynamical model considers the effects of tooth surface roughness changes and geometric deviations due to pitting and spalling, and also incorporates Time Varying Mesh Stiffness (TVMS), a time-varying load sharing ratio, as well as dynamic tooth contact friction forces, friction moments and dynamic mesh damping ratios. The proposed gear dynamical model is validated by comparison with responses obtained from experimental test rig under different conditions. Comparisons indicate that the responses of the proposed dynamical model are consistent with experimental results, in both time and frequency domains under different rotation speeds and fault severity conditions.