Rechargeable magnesium batteries (RMBs) are promising candidates to replace currently commercialized lithium‐ion batteries (LIBs) in large‐scale energy storage applications owing to their merits of abundant resources, low cost, high theoretical volumetric capacity, etc. However, the development of RMBs is still facing great challenges including the incompatibility of the electrolyte and the lack of suitable cathode materials with high reversible capacity and fast kinetics of Mg2+. While tremendous efforts have been made to explore compatible electrolytes and appropriate electrode materials, the rational design of unconventional Mg‐based battery systems is another effective strategy for achieving high electrochemical performance. This review specifically discusses the recent research progress of various Mg‐based battery systems. First, the optimization of electrolyte and electrode materials for conventional RMBs is briefly discussed. Furthermore, various Mg‐based battery systems, including Mg‐chalcogen (S, Se, Te) batteries, Mg‐halogen (Br2, I2) batteries, hybrid‐ion batteries, and Mg‐based dual‐ion batteries are systematically summarized. This review aims to provide a comprehensive understanding of different Mg‐based battery systems, which can inspire latecomers to explore new strategies for the development of high‐performance and practically available RMBs. This review specifically presents current progress on recently developed rechargeable magnesium batteries, including Mg‐chalcogen batteries, Mg‐halogen batteries, hybrid ion batteries, and dual‐ion batteries. Additionally, challenges and potential solutions for these battery systems are proposed. This review aims to show new strategies for the development of high‐performance and practically available magnesium batteries.