One of the key quality aspects in a probabilistic prediction is its reliability. However, this property is difficult to estimate in the case of seasonal forecasts due to the limited size of most of the hindcasts that are available nowadays. To shed light on this issue, this work presents a detailed analysis of how the ensemble size, the hindcast length and the number of points pooled together within a particular region affect the resulting reliability estimates. To do so, we build on 42 land reference regions recently defined for the IPCC‐AR6 and assess the reliability of global seasonal forecasts of temperature and precipitation from the European Center for Medium Weather Forecasts SEAS5 prediction system, which is compared against its predecessor, System4. Our results indicate that whereas longer hindcasts and larger ensembles lead to increased reliability estimates, the number of points that are pooled together within a homogeneous climate region is much less relevant. Plain Language Summary Seasonal climate forecasts provide information on the average conditions that can be expected for the next months (up to a year) and can help decision making in different socio‐economic sectors such as agriculture, energy and health (among others). However, predictability at this time‐scale is in general limited, so the actual usefulness of seasonal forecasts must be carefully evaluated before they are used in practical applications. In this aspect, reliability—which measures how well/bad the forecast probability for a particular event fits with its actual occurrence—is a key property. This work assesses the reliability of global seasonal forecasts of temperature and precipitation using the latest operational seasonal forecasting system from European Center for Medium Weather Forecasts. Our results show that reliability is generally better for temperature than for precipitation. Moreover, we demonstrate that reliability is sensitive to the number of retrospective forecasts (known as hindcast) and ensemble members (from which forecast probabilities are obtained) available. Finally, we also demonstrate that the new IPCC‐AR6 land reference regions are adequate for seasonal verification purposes. These findings are important for a fair interpretation of the reliability of seasonal forecasts which are obtained for specific regions/seasons/systems building on different experimental frameworks. Key Points KP1 The new SEAS5 from the European Center for Medium Weather Forecasts increases the reliability of the previous System4 for global seasonal predictions of temperature and precipitation KP2 The reliability of probabilistic seasonal forecasts can vary substantially due to the ensemble size and the length of the available hindcast KP3 The newly defined IPCC‐AR6 land reference regions are adequate for the verification of seasonal forecast reliability