Rumor is defined as an unconfirmed public opinion or gossip, which can be propagated through various channels, causing serious economic losses and disturbing the normal order of life. Therefore, studying the rumor propagation mechanism is helpful to implement effective rumor propagation control strategies. In this paper, the wise man is involved in the dynamic process of rumor propagation, and the modified rumor propagation model is established with consideration of social reinforcement and forgetting factors in heterogeneous network. The theories analysis of the model is carried out based on the mean field equation. The rumor propagation threshold is obtained, which is closely related to dual social reinforcement, wisdom conversion rate and forgetting factors. The final size of the rumor is related to positive social reinforcement and negative social reinforcement. Numerical simulations indicate that, the social positive reinforcement promotes rumor propagation and social negative reinforcement suppresses rumor propagation. And the addition of wise men reduces the number of spreaders, thus affecting the process of rumor propagation. •We firstly introduce a modified ISWR rumor propagation model on heterogeneous networks.•We derive the mean field equations with considering the wise man and social reinforcement.•The wisdom conversion rate and the social reinforcement reshapes the critical threshold of spreading dynamics.•The social reinforcement brings a nonlinear impact on propagation speed and final size of the rumor propagation.