Accurate and reliable monthly runoff forecasting plays an important role in making full use of water resources. In recent years, long short-term memory neural networks (LSTM), as a deep learning technology, has been successfully applied in forecasting monthly runoff. However, the hyperparameters of LSTM are predetermined, which has a significant influence on model performance. In this study, given that the decomposition of monthly runoff series may provide a more accurate prediction, as revealed by many previous studies, a hybrid model, namely, VMD-GWO-LSTM, is proposed for monthly runoff forecasting. The proposed hybrid model comprises two main components, namely, variational mode decomposition (VMD) coupled with the gray wolf optimizer (GWO)-based LSTM. First, VMD is utilized to decompose raw monthly runoff series into several subsequences. Second, GWO is implemented to optimize the hyperparameters of the LSTM for each subsequence on the condition that the inputs are determined. Finally, the total output of all subsequences is aggregated as the final forecast result. Four quantitative indices are employed to evaluate the model performance. The proposed model is demonstrated using 73 and 62 years of monthly runoff series data derived from the Xinfengjiang and Guangzhao Reservoirs in China's Pearl River system, respectively. To identify the feasibility and superiority of the proposed model, backpropagation neural networks (BPNN), support vector machine (SVM), LSTM, EMD-LSTM, VMD-LSTM and GWO-LSTM are also utilized for comparison. The results indicate that the proposed hybrid model can yield best forecast accuracy among these models, making it a promising new method for monthly runoff forecasting.