•We present a coal-gas outburst hypothesis model, which describes the progressive failure process of gas-bearing coal based on strength reduction theory.•Considering gas adsorption and seepage, the yield failure criterion and limit instability criterion of gas-bearing coal are established respectively.•Simulate the fluid-structure coupling characteristics under strength reduction.•Then the correlation between (limit) safety factor and outburst risk is explained, and verified by laboratory tests. Coal is a kind of natural pore-fracture dual medium, which determines the diffusion and seepage characteristics of gas inside under the stress conditions. In turn, the gas migration has an effect on the mechanical damage properties of coal and is closely related to the occurrence of coal-gas outburst. The occurrence of outburst is inevitably accompanied by the instantaneous displacement mutation of gas-bearing coal, which is difficult to explained by conventional plastic zone analysis. And the process from plastic failure to instability is still vague, which makes the instability critical of outburst difficult to determine. Therefore, aiming at the outburst in the working face, we put forward a progressive failure and outburst hypothesis model based on the limit analysis method of strength reduction theory. Then, the limit instability criterion of gas-bearing coal is modified, and the coal-gas outburst in the strength reduction process is simulated by using the cyclic calculation method of COMSOL software. It describes the characteristics of fluid–structure coupling and progressive failure instability of gas-bearing coal. Finally, we use the limit safety factor to describe the outburst criticality and the outburst risk. We find that the initial uncovering coal area, stress and gas pressure play an important role in controlling outburst, especially the coal stability is extremely sensitive to gas pressure changes under high stress. And based on these, the outburst prevention and control measures are put forward. The method has potential application value for the risk assessment and prevention of outburst on site.