This paper describes an optimal design of a notch filter for the active magnetic bearing (AMB) system which has unstable poles. A Notch filter is designed as 2 nd order standard transfer function with two zeros and two poles. We performed optimization to design a notch filter using sequential quadratic programming (SQP) and evaluate the closed loop stability with a designed notch filter. The designed notch filter is optimized to have the lowest peak sensitivity with respect to the disturbance closed loop system. The designed notch filter is examined for the compensation effect for the unstable 2 nd order system. We applied the design optimization to large scale flywheel energy storage system (FESS). FESS can store up to 5kWh of usable energy at the maximum speed of 18,000 rpm. The effect of the designed filter from optimization is simulated using simulation model of FESS and the simulation model is validated against experiments.