Demand for quantitative assessments of likely climate change impact on runoff is increasing and conceptual rainfall‐runoff models are essential tools for this task. However, the capacity of these models to extrapolate under changing climatic conditions is questionable. A number of studies have found that model predictive skill decreases with changed climatic conditions, especially when predicting drier climates. We found that model skill only declines under certain circumstances, in particular, when a catchment's rainfall‐runoff processes change due to changed climatic drivers. In catchments where the rainfall‐runoff relationship changed significantly in response to prolonged dry conditions, runoff was consistently overestimated. In contrast, modeled runoff was unbiased in catchments where the rainfall‐runoff relationship remained unchanged during the dry period. These conclusions were not model dependent. Our results suggest that current projections of runoff under climate change may provide overly optimistic assessments of future water availability in some regions expecting rainfall reductions. Key Points Hydrological model performance often degrades during prolonged shifts in climate Climate shifts sometimes lead to changes in internal catchment functioning Models perform poorly and become strongly biased where such changes occur, but not otherwise