•Physical origin of cyclic softening discrepancy between stress and strain cycling are clarified.•A cyclic softening model is developed involving the special dislocation annihilation and storage events.•Various cyclic responses under different loading modes were predicted very well by the proposed model with a single parameter set. Stress and strain controlled low cycle fatigue of the modified 9–12% Cr steel with the hierarchical arranged lath martensitic structure were conducted. We found that, apart from microstructure recovery in lath structure, an additional mechanism of reverse avalanche of low angle boundary, associated with the burst-like plastic deformation is responsible for the accelerated softening during stress cycling. These microstructural evolutions can be explicitly represented via deriving different dislocation evolution laws in terms of dislocation annihilation and storage events. Accordingly, a new model is proposed involving the microstructural evolution. Results indicate that the accelerated softening behavior under stress cycling can be reproduced very well by the present model with a very limited number of adjustable parameters. In addition, the model can capture the features of cyclic response and microstructural evolution under both the strain and stress cycling over a wide range of amplitudes.