Objectives To reveal distinct longitudinal trajectories in episodic memory over 15 years and to identify demographic, lifestyle, health‐related, and genetic predictors of stability or decline. Design Prospective cohort study. Setting The Betula Project, Umeå, Sweden. Participants One thousand nine hundred fifty‐four healthy participants aged 35 to 85 at baseline. Measurements Memory was assessed according to validated episodic memory tasks in participants from a large population‐based sample. Data were analyzed using a random‐effects pattern‐mixture model that considered the effect of attrition over two to four longitudinal sessions. Logistic regression was used to determine significant predictors of stability or decline relative to average change in episodic memory. Results Of 1,558 participants with two or more test sessions, 18% were classified as maintainers and 13% as decliners, and 68% showed age‐typical average change. More educated and more physically active participants, women, and those living with someone were more likely to be classified as maintainers, as were carriers of the met allele of the catechol‐O‐methyltransferase gene. Less educated participants, those not active in the labor force, and men were more likely to be classified as decliners, and the apolipoprotein E ɛ4 allele was more frequent in decliners. Conclusion Quantitative, attrition‐corrected assessment of longitudinal changes in memory can reveal substantial heterogeneity in aging trajectories, and genetic and lifestyle factors predict such heterogeneity.