Electric vehicles (EVs) are playing an important role in power systems due to their significant mobility and flexibility features. Nowadays, the increasing penetration of renewable energy resources has been observed in modern power systems, which brings many benefits for improving climate change and accelerating the low-carbon transition. However, the intermittent and unstable nature of renewable energy sources introduces new challenges to both the planning and operation of power systems. To address these issues, vehicle-to-grid (V2G) technology has been gradually recognized as a valid solution to provide various ancillary service provisions for power systems. Many studies have developed model-based optimization methods for EV dispatch problems. Nevertheless, this type of method cannot effectively handle the highly dynamic and stochastic environment due to the complexity of power systems. Reinforcement learning (RL), a model-free and online learning method, can capture various uncertainties through numerous interactions with the environment and adapt to various state conditions in real-time. As a result, using advanced RL algorithms to solve various EV dispatch problems has attracted a surge of attention in recent years, leading to many outstanding research papers and important findings. This paper provides a comprehensive review of popular RL algorithms categorized by single-agent RL and multi-agent RL, and summarizes how these advanced algorithms can be applied to various EV dispatch problems, including grid-to-vehicle (G2V), vehicle-to-home (V2H), and V2G. Finally, key challenges and important future research directions are discussed, which involve five aspects: (a) data quality and availability; (b) environment setup; (c) safety and robustness; (d) training performance; and (e) real-world deployment. Display omitted •State-of-the-art reinforcement learning algorithms are reviewed.•Both single-agent and multi-agent reinforcement learning frameworks are discussed.•Three key applications of reinforcement learning to EV dispatch problems are reviewed.•Challenges and future directions of applying reinforcement learning to EV problems are discussed.