As cognitive function declines with age, identifying factors affecting the trajectory of cognitive decline is an indispensable step toward developing intervention strategies to improve the quality of the elderly life. We performed a genome-wide association study (GWAS) focusing on memory function to explore single nucleotide polymorphisms (SNPs) associated with the rate of memory decline. Seven hundred and nine eligible non-Hispanic Caucasians from the Alzheimer's Disease Neuroimaging Initiative (ADNI) were included for analysis after quality control. GWAS was performed with linear regression. We subsequently tested whether the associations remained significant in subgroup analysis and also examined the impact of SNPs on the longitudinal changes in other neuropsychological measures and amyloid pathology. We identified rs13374761-A in SLAMF1 gene associated with less memory decline (MAF = 0.071, β= 0.0103, p = 4.14×10-8). Subgroup analysis showed stability of results across groups with different diagnosis at baseline. Rs13374761-A also had protective effects on global cognition (p = 0.024), episodic memory (p = 0.024), and semantic memory (p = 0.042), and exerts protection against a decrease in CSF Aβ42 concentration (p = 0.0463) and an increase in Aβ loading in cerebral cortex (p = 0.00666) among minor allele carriers. A novel variant in gene SLAMF1 affects the rate of memory decline in the aged population. Given the protective effect of this variant, SLAMF1 should be further investigated as a potential preventive and therapeutic target for monitoring cognition trajectories.