Aiming at the complexity, nonlinearity and difficulty in modeling of nonlinear system. In this paper, an improved back-propagation(BP) neural network based on restricted boltzmann machine(RBM-IBPNN) is proposed for nonlinear systems modeling. First, the structure of BP neural network(BPNN) is optimized by using sensitivity analysis(SA) and mutual information(MI) of the hidden neurons. Namely when the SA value and the MI value of the hidden neurons satisfy the set standard, the corresponding neurons will be pruned, split or merged. second, the restricted boltzmann machine(RBM) is employed to perform parameters initialization of training on the IBPNN. Finally, the proposed RBM-IBPNN is evaluated on nonlinear system identification, lorenz chaotic time series prediction and the total phosphorus prediction problems. The experimental results demonstrate that the proposed RBM-IBPNN not only has faster convergence speed and higher prediction accuracy, but also realizes a more compact network structure.