Complex mechanical systems, such as wind turbines, often include safety constraints which should not be violated in order to avoid high risk of structural damage. Adherence to the safety constraints is ensured with a well‐designed operating envelope protection. In this paper, we present an invariant set‐based protection concept with the application to the wind turbine overspeed protection. The approach is model‐based and operates without wind‐preview measurements. Instead, it is based on the evolution model of the wind disturbance in the worst‐case scenario manner. Accurate system estimation utilizing the blade root in‐plane measurement is proposed, along with the efficient algorithm for the hard real‐time operation of the protection system. The overspeed protection system is validated in extensive simulations on extreme turbulent wind, performed in GH Bladed. The results show the effectiveness of the proposed concept compared with the baseline controller performance. Copyright © 2015 John Wiley & Sons, Ltd.