The synchrosqueezing transform can effectively improve the readability of time–frequency representation of mono-component and constant frequency signals. However, for multi-component and time-variant frequency signals, it still suffers from time–frequency blurs. In order to address this issue, the synchrosqueezing transform is improved using iterative generalized demodulation. Firstly, the complex nonstationary signal is decomposed into mono-components of constant frequency by iterative generalized demodulation. Then, the instantaneous frequency of each mono-component is accurately estimated via the synchrosqueezing transform, by exploiting its merit of enhanced time–frequency resolution. Finally, the time–frequency representation of the original signal is obtained by superposing the time–frequency representations of all the mono-components with restored instantaneous frequency. This proposed method generalizes the synchrosqueezing transform to multi-component and time-variant frequency signals, and it has fine time–frequency resolution and is free of cross-term interferences. The proposed method was validated using both numerically simulated and lab experimental vibration signals of planetary gearboxes under nonstationary conditions. The time-variant planetary gearbox characteristic frequencies were effectively identified, and the gear faults were correctly diagnosed. •The proposed method has fine time–frequency resolution and is free from cross term interferences.•It is suited to analyze signals of multi-component and time-varying frequency.•It is evaluated using numerical simulated and lab experimental planetary gearbox signals.•It is effective in diagnosing both the local and distributed gear faults.