Forecasting crowd congestion is a critical aspect of crowd management, particularly in dynamic and densely populated areas, such as urban centers, events, or pilgrimage sites. In this paper, we proposed the first crowd congestion forecasting framework for the pilgrimage of Umrah. We addressed the crowd congestion forecasting problem by clustering the crowd flow trajectory in Masjid Al-Haram (Great Mosque) in the city of Makkah into six zones. The framework consists of two main components: 1) Ensemble forecasting model that aims at forecasting the crowd density of Masjid Al-Haram and its six zones, and 2) decision making algorithm that aims at keeping the crowd density at an acceptable level, and recommends updating the crowd flows when the forecasted crowd density exceeds the crowd density threshold. We built the ensemble learning model in three phases. In the first phase, we selected and evaluated different learning base models, including ARIMA, Sequence to Sequence (Seq2Seq) learning, M-1D-CNN-LSTM, and DeepSTN. In the second phase, the best three models, which performed well in the first phase, are selected to build the stacked ensemble model. The latter is validated using the walk-forward technique in the third phase. To evaluate the framework, we built a crowd dataset based on two temporal properties: 1) hourly context and 2) daily context. We evaluated the three phases of the ensemble forecasting model. In the first phase, DeepSTN performs the best by achieving a Mean Absolute Error (MAE) of 0.281. The results also indicate that DeepSTN is the best fit for five zones, and one variant of Seq2Seq, named Seq2Seq2b is the best fit for one zone under Mean Square Error (MSE) and Root Mean Squared Error (RMSE). Under MAE, DeepSTN and Seq2Seq2b, each of which is the best choice for three zones. In the second phase, the stacked ensemble achieves a MAE of 0.257. In the third phase, the stacked ensemble model is validated using the walk forward technique, which allows to reduce the MAE to 0.253. Although this framework focuses on Umrah, it can be customized for other use cases that involve crowd congestion forecasting.