Electricity plays a crucial role in the well-being of humans and is a determining factor of the economic development of a country. Electricity issues have encouraged researchers to focus on improving power availability and quality along with reliability. This pursuit has increasingly raised the intention to integrate renewable energy (RE) into power systems to curb the problem of energy deficiency. However, intermittency in the sources of RE supply coupled with fluctuating changes in demand with respect to time has induced high risk in maintaining system reliability in terms of providing adequate supply to consumers. Whilst an energy storage system (ESS) is not another source of electricity, it is proven to be effective and viable in solving the aforementioned issues. Thus, this paper comprehensively reviews the development of ESS technologies and discusses the benefits and real-life applications of these technologies. The concept of reliability in power systems is also explored to provide an improved understanding of this study. Lastly, notable studies that have addressed the reliability impact of ESSs on power systems are discussed. This review paper therefore is expected to provide a critical analysis of ESS developments, as well as recognize their research gaps in terms of reliability studies in modern RE-integrated power networks.