Background: Information concerning longitudinal cognitive trajectories in multiple sclerosis (MS) is relatively scarce. Moreover, it is unclear which factors are associated with cognitive decline and what is the clinical impact of cognitive impairment (CI) in the long run. Objective: To investigate cognitive trajectories in relapsing–remitting multiple sclerosis (RRMS) patients, analyzing clinical and magnetic resonance imaging (MRI) predictors of cognitive decline. Methods: We enrolled 42 patients and 30 controls. They underwent brain MRI and clinical/neuropsychological evaluation at baseline and after 1, 2, and 6 years. We evaluated cognitive domains with principal component analysis and performed multivariable regression analyzing predictors of clinical/cognitive deterioration. We also performed repeated measures analysis to assess whether clinical progression was different according to CI at baseline. Results: A total of 23 (62.2%) patients deteriorated in combined cognitive domains after 6 years, most in processing speed and memory. The number of baseline impaired cognitive domains was strongly associated with 6-year cognitive (R2 = 0.452; p < 0.001) and Expanded Disability Status Scale (EDSS) deterioration (R2 = 0.263; p < 0.001). Patients with baseline CI in combined domains had worse clinical progression. Conclusion: Isolated CI tends to become more widespread, affecting memory and processing speed alongside. The extent of baseline CI was the best predictor of both clinical and cognitive deterioration after 6 years.