•An innovative self-centering (SC) modular steel structure (MSS) connection is proposed.•Seismic behaviors of MSS and SC-MSS connections are investigated due to cyclic loadings.•SC-MSS connection exhibits better bearing capacity and self-centering capability.•SC haunch braces can reduce the plastic damage and improve the ductility of connection. As an efficient and environment-friendly technique that combines off-site manufacturing of modules and on-site assembly, modular buildings have been adopted in many areas of the construction industry. This paper proposes an innovative modular steel structure (MSS) connection with bolted cover plates, which inserts a cross-shaped plug-in connector inside the adjacent columns to achieve accurate positioning in the module installation and provide partial shear capacity. A seismic retrofit method using hinged diagonal self-centering (SC) haunch braces is proposed for the MSS connection to improve the seismic performance and achieve functional recoverability after earthquakes. Hysteretic behaviors of the MSS connection and the self-centering MSS (SC-MSS) connection are investigated due to cyclic loadings. Results demonstrate that compared with the MSS connection, the SC-MSS connection is expected to improves strength and stiffness and exhibits stable flag-shaped hysteretic responses with a satisfactory self-centering capability. The hinged diagonal SC haunch braces are capable of reducing the average stress level around the connection core region, controlling the development of plastic damage, avoiding the local failure, and improving the ductility of connection. The present study provides practical connections and important guidance for the applications of modular steel structures in the medium- and high-intensity seismic regions.