Model predictive control is a promising approach to reduce the CO2 emissions in the building sector. However, the vast modeling effort hampers the widescale practical application. Here, data-driven process models, like artificial neural networks, are well-suited to automatize the modeling. However, the underlying data set strongly determines the quality and reliability of artificial neural networks. In general, the validity domain of a machine learning model is limited to the data that was used to train it. Predictions based on system states outside that domain, so-called extrapolations, are unreliable and can negatively influence the control quality. We present a safe operation approach combined with online learning to deal with extrapolation in data-driven model predictive control. Here, the k-nearest neighbor algorithm is used to detect extrapolation to switch to a robust fallback controller. By continuously retraining the artificial neural networks during operation, we successively increase the validity domain of the artificial neural networks and the control quality. We apply the approach to control a building energy system provided by the BOPTEST framework. We compare controllers based on two data sets, one with extensive system excitation and one with baseline operation. The system is controlled to a fixed temperature set point in baseline operation. Therefore, the artificial neural networks trained on this data set tend to extrapolate in other operating points. We show that safe operation in combination with online learning significantly improves performance. Display omitted •We introduce safe operation for online learning data-driven model predictive control.•Novelty Detection is used to check the reliability of artificial neural networks.•Continuously retraining the process models and detector increases the validity domain.•The methodology achieves significant improvements when only poor initial data is available.