Development of proton exchange membranes with sufficiently high proton conductivity, especially at low relative humidity (RH), remains a big challenge in the field of fuel cells. In this study, graphene oxide-based nanoscale ionic materials (NIMs-GO) were prepared by sulfonation with 3-(trihydroxysilyl)-1-propane­sulfonic acid and subsequent neutralization with amino-terminated polyoxypropylene (PO)–polyoxyethylene (EO) block copolymer. The resultant NIMs-GO with acid–base pairs and hygroscopic EO units were incorporated into sulfonated polysulfone (SPSF) to fabricate nanocomposite membranes. A matrix-softening phenomenon was found due to the extensive interaction between the SPSF matrix and the amphiphilic NIMs-GO, which primarily contributes to the homogeneous dispersion of the NIMs-GO filler in the nanocomposite membranes. The acid–base pairs and the interconnected hydrogen-bonded networks formed between the EO units and water molecules imparted efficient proton transfer via the Grotthuss mechanism. The water uptake and retention ability of the SPSF/NIMs-GO nanocomposite membranes were enhanced due to the hydrophilic EO units on NIMs-GO. As a result, the nanocomposite membrane exhibited a 52% increase compared with the pristine SPSF membrane in proton conductivity at 75 °C, 100% RH and a 24-fold increase at 75 °C, 40% RH. This enhanced proton conductivity led to an elevated fuel cell performance under both hydrous and low RH conditions.