In a power system with high renewable power penetration, uncertain and intermittent characteristics of renewable energy resources become major concerns of system planning and operation. In such a power system, conventional thermal or hydro units have to be dispatched, shut down or started up more frequently to incorporate load, renewable resource variation and system failures. Slow ramp up (or down) of conventional units may cause energy shortage (or surplus) in power systems with high wind power penetration or in isolated micro grids with limited dispatchable capacity. This paper proposes a technique to evaluate operational reliability and energy utilization efficiency of power systems with high wind power penetration. The ramp rate of a conventional generator and energy storage system (ESS) are considered in the proposed technique. The effect of slow ramp-up rate or fast reduction of wind speed on system reliability is measured by the expected energy not supplied. A new index designated as the expected energy not used is proposed and formulated to represent energy surplus due to fast increase of wind speed and slow ramp down of conventional units. The IEEE-RTS is used to illustrate the applications of the proposed method.