Despite the well-documented detrimental health effects of elder abuse, scholars have rarely considered whether and how family members’ abuse of older adults is associated with sleep. Data from the 2018 China Longitudinal Aging Social Survey (N = 8110) were used to assess the association between elder abuse committed by family members and sleep quality, and how psychological well-being (depression and loneliness) mediates and gender moderates the above association. Results indicated that those who experienced family members’ abuse were more likely to report poor sleep quality than their non-abused counterparts. Depression and loneliness partially mediated the elder abuse–sleep relationship. Furthermore, among those who experienced one elder abuse, older women had a higher risk of poor sleep quality than their male counterparts. However, two or more elder abuse experiences had similar negative effects on older women and men. Preventing elder abuse and improving psychological well-being is critical to promoting late-life sleep.