This paper presents a modified incremental harmonic balance (IHB) method to analyze the nonlinear dynamic and thermal coupling characteristics of rotor-bearing systems after thermal balance. The IHB method is modified by tensor contraction and fast Fourier transform (FFT) to efficiently calculate the residual vector and Jacobian matrix. Nonlinear dynamic and thermal full coupling is implemented through the nonlinear part of the Jacobian matrix. The presented method achieves the simultaneous and efficient calculation of the motion and heat balance equations in the frequency domain according to the improvement measures. The effectiveness of the presented method is tested using the vibration response and temperature variation analysis for the coupled thermo-mechanical model of a rotor-ball bearing system. Furthermore, the performance comparisons show the good consistency of results between the presented method and the step-by-step (S-S) method, which is a partitioned iterative solution method for dealing with fully coupled problems in the time domain. The presented method has greater computational efficiency, more accurate temperature prediction, and better recognition of nonlinear phenomena in the system rather than the S-S method. Consequently, the presented method has a broad application prospect in solving fully coupled thermo-mechanical problems of more complex and higher dimensional rotor-bearing systems. •The motion and heat balance equations can be calculated simultaneously.•Greater calculation efficiency is illustrated than the step-by-step method.•Temperature can be predicted more accurately by time-varying nonlinear forces.•Nonlinear phenomenon recognition of coupled thermo-mechanical problems is improved.