•A M-band flexible wavelet transform with high frequency resolution is constructed.•The wavelet filter bank is designed by the frequency domain method.•The fast numeric implementation algorithm of the proposed transform is presented.•This method can be well applied to diagnose the faults of planetary gearboxes. The fault diagnosis of planetary gear transmission systems is crucial for the safety of machineries and equipment. To identify the underlying fault features in measured signals, a novel M-band flexible wavelet transform is constructed. This transform provides a denser sampling of the time–frequency plane and preserves tunable filter parameters and dilation factors. A perfect reconstruction condition of the proposed transform is established, and its corresponding wavelet filter bank is designed to satisfy the perfect reconstruction condition. A numerical implementation algorithm of M-band flexible wavelet transform is investigated using a multirate filter bank and fast Fourier transform. Denoising the simulation signals demonstrates that the proposed transform exhibits better performance than analytic flexible wavelet transform, orthogonal wavelet transform, and biorthogonal wavelet transform. A new fault diagnosis method for planetary gear transmission systems is proposed on the basis of M-band flexible wavelet transform and spectral negentropy. Experimental and comparative results show that the proposed method can be more effectively and accurately applied to the fault diagnosis of planetary gear transmission systems compared with typical fault diagnosis methods based on analytic flexible wavelet transform, Morlet wavelet transform, and infograms.