•A modified HB-AFT method is proposed, which can solve high-dimensional differential equations with complex nonlinearities programmatically and efficiently.•All periodic solutions of a dual-rotor-bearing-casing system are obtained, the nonlinear dynamics of the system is analyzed.•The nonlinear dynamic behaviors such as vibration jumping, bi-stable and resonance hysteresis are revealed.•The evolution of the inter-shaft bearing’s nonlinear dynamic load is analyzed, jumping phenomena are also found here. This paper presents a modified harmonic balance-alternating frequency/time domain (HB-AFT) method to analyze the nonlinear dynamics of a dual-rotor-bearing-casing system. The motion equations of a dual-rotor-bearing-casing system subjected to the unbalanced excitations of the two rotors are formulated with 284 degrees of freedom, in which, the complex nonlinear factors of the inter-shaft bearing such as exponential nonlinear restoring force, radial clearance and varying stiffness excitation are taken into consideration. The HB-AFT method is modified by applying the AFT procedure to obtain the harmonic expending coefficients of the harmonic balance residuals rather than that of the nonlinear terms, hence, the Jacobian matrix is calculated programmatically and efficiently, as a result the performance of the HB-AFT method is improved greatly. The primary resonance of the system is analyzed by employing the three-dimensional diagrams of amplitude-frequency responses of each node of the system, including the inner casing and the outer casing. In addition, the amplitude-frequency response curves with separated frequencies and orbits of the two rotors are also discussed. Furthermore, the pseudo-arclength continuation procedure is introduced into the modified HB method, then all periodic solution branches of the system including the unstable solutions are obtained. The results show that there are two resonant peaks in the amplitude-frequency response curves of the system and the vibration of each part of the system, i.e., LPC, LPT, HPC, HPT, inner casing and outer casing, is synchronous. Besides, the nonlinear dynamic behaviors such as vibration jumping, bi-stable and resonance hysteresis are revealed. Accordingly, the dynamic load of the inter-shaft bearing is calculated. The results show that the evolution of the inter-shaft bearing’s dynamic load is associated with the vibration responses of the two rotors. The nonlinear characteristics of the force-frequency responses of the inter-shaft bearing are consistent with that of the two rotors and casings. In addition, with the increase of the inter-shaft bearing clearance, the nonlinear characteristics of the force-frequency responses of the inter-shaft bearing are more significant. In comparison with the Newmark method, the modified HB-AFT method proposed in this paper has a great advantage in computing efficiency, and it can also grasp all solution branches of the system including the unstable solutions. Consequently, it has great potential to deal with high-dimensional systems with complex nonlinearities and multifrequency excitations in practical engineering.