Display omitted •A novel aseismic system – hybrid self-centering rocking core system - was proposed.•A performance-based design method is developed for the HSRC system.•The HSRC systems show excellent performance in controlling structural damage and have enough redundancy to resist building collapse under extreme earthquakes.•The HSRC systems can show excellent recoverability against extreme earthquakes.•The HSRC systems can show credible performance in controlling nonstructural damage. This research presents a novel aseismic system, the hybrid self-centering rocking core (HSRC) system, for obtaining better seismic resilience in steel buildings. Two hybrid self-centering dampers are introduced in the HSRC system to control structural and nonstructural damage. A rocking core, i.e., a steel braced frame with pinned base is included in the HSRC system to facilitate uniform inter-story drift responses under earthquakes. The steel braces in the rocking core can also work as a reserve energy-absorbing mechanism against structural collapse subjected to extreme seismic events whose intensities are more significant than those of the maximum considered earthquakes in design codes. The desired nonlinear responses of the HSRC system were analyzed. Direct displacement-based design steps were developed for the HSRC system. Three-story and six-story HSRC systems were designed following the proposed design method. Nonlinear dynamic analysis results indicate that the designed HSRC systems can show desired nonlinear responses and achieve the expected performance objective under earthquakes. Moreover, seismic fragility analyses were also conducted based on the incremental dynamic analysis results to assess the performance in controlling structural and nonstructural damages of the HSRC system under earthquakes of different intensities. The results confirm that the proposed HSRC system has excellent capacity in resisting structural collapse, achieving reliable post-earthquake recoverability, and controlling structural and nonstructural damage under rare seismic events.