Through analyzing the characteristics of the Miner’s rule and the Manson-Halford (M-H) model, a nonlinear fatigue damage accumulation model is established based on the damage curve method under combined high and low cycle fatigue loading. In this model, an interaction factor related to the maximum stress and high cycle stress range is proposed to consider the loading interaction effect. Furthermore, the proposed model does not require any additional material parameters except for the S-N curve parameters. The experimental data of five materials, including TC11 alloy, 2024-T3 aluminum alloy, 45 steel, LY12-CZ aluminum alloy, and GH4033 Ni-based alloy, are introduced to verify the proposed model. Compared with the Miner’s rule and the M-H model, most of the prediction results by the proposed model are within the life factor range of 2, implying the established model has higher prediction accuracy.