To understand the differences in affective memory performance under different degrees of cognitive impairment, this study recruited older people with different degrees of cognitive impairment, to perform emotion recognition memory tasks. Fifty-four elderly participants aged (65-85 years) were recruited. Of these, 18 had mild cognitive impairment, 18 had a mild form of Alzheimer's disease, and the remaining 18 were healthy. Factors such as the different emotional valences (positive, neutral, or negative) and stimulus types (pictures, words, or sounds) were manipulated to explore their influences on the emotion recognition memory of people with different degrees of cognitive impairment. The results showed that people's performance to positive stimuli worsened as their degree of cognitive impairment increased. All participants had difficulty processing memory of affective sound stimuli compared to the other two stimulus types. The results explain the decline in the cognitive ability process, in affective memory performance, of people with different degrees of cognitive impairment. This abnormal decline on affective memory performance could be an early diagnostic indicator of Alzheimer's disease. The results can hopefully be used as a reference for subsequent research on cognition-related diseases and age-related decline, especially regarding affective memory.