Despite the growing interest in the effects of neighborhood environments on cognitive function, most studies on older people are based on cross-sectional survey data from developed countries. This study examines the relationship between neighborhood environments and decline in cognitive function over time among middle-aged and older people in China and whether this relationship varies between rural and urban residents. The three waves of China Health and Retirement Longitudinal Study (CHARLS 2011-2015) were used. The sample included 12,131 respondents living in 298 rural villages and 4,059 respondents living in 150 urban communities. Three-level linear growth curve models were estimated to track trajectories of cognitive change over a 4-year period. Chinese older people who lived in neighborhoods with more handicap access, more bus lines, employment service, and higher socioeconomic status (SES) had slower cognitive decline. Neighborhood basic infrastructures, number of days that roads were unpassable, outdoor exercise facilities, and average social activity participation were associated with baseline cognitive function in both rural and urban areas, but neighborhood environments had more impact on cognitive decline among rural older adults than urban older adults. Findings from this study call for increased infrastructure development and community building programs in rural China.