A thorough understanding of the spectral structure of planetary gear system vibration signals is helpful to fault diagnosis of planetary gearboxes. Considering both the amplitude modulation and the frequency modulation effects due to gear damage and periodically time variant working condition, as well as the effect of vibration transfer path, signal models of gear damage for fault diagnosis of planetary gearboxes are given and the spectral characteristics are summarized in closed form. Meanwhile, explicit equations for calculating the characteristic frequency of local and distributed gear fault are deduced. The theoretical derivations are validated using both experimental and industrial signals. According to the theoretical basis derived, manually created local gear damage of different levels and naturally developed gear damage in a planetary gearbox can be detected and located.