The aim of this paper is to investigate a novel approach for output feedback damping controller design of the static synchronous compensator (STATCOM) in order to enhance the damping of power system low frequency oscillations (LFO). The design of output feedback controller is considered as an optimization problem according to the time domain-based objective function solved by a honey bee mating optimization (HBMO) algorithm that has a strong ability to find the most optimistic results. To validate the accuracy of results a comparison with genetic algorithm (GA) has been made. The effectiveness of the proposed controller are tested and demonstrated through eigenvalue analysis and nonlinear time-domain simulation studies over a wide range of loading conditions. The simulation study shows that the designed controller by HBMO performs better than GA in finding the solution.