Background The association between sarcopenia and mild cognitive impairment (MCI) among elderly adults in China remains unclear. The present study aimed to examine the association based on a nationally representative large‐scale survey. Methods The study used two waves of data from China Health and Retirement Longitudinal Study (CHARLS) in 2015 and 2018. All subjects met the inclusion criteria were classified based on Asia Working Group for Sarcopenia 2019 criteria. Aging‐associated cognitive decline is used to define MCI, and cognitive function is measured based on four dimensions: orientation, computation, memory, and drawing. OLS and logistic regression model were conducted to analyse the cross‐sectional association between sarcopenia and different cognitive functions. Logistic regression model was conducted to analyse the longitudinal association between sarcopenia and MCI. Results Totally, 5715 participants aged over 60 years (43.8% women; mean age 67.3 ± 6.0 years) were enrolled in a cross‐sectional association study in 2015, and further 2982 elderly adults were followed up in 2018. During the period, sarcopenia and possible sarcopenia increased from 8.5% to 29.6%. Scores of cognitive and four dimensions (orientation, computation, memory, and drawing) exhibited a decreasing trend from non‐sarcopenia to sarcopenia (P < 0.001). In the fully adjusted OLS regression model, scores of four dimensions were lower in possible sarcopenia and sarcopenia groups when compared with the non‐sarcopenia group (P < 0.05) respectively. The incidence of MCI was 10.1%, 16.5%, and 24.2% for non‐sarcopenia, possible sarcopenia, and sarcopenia groups from 2015 to 2018, with a significantly statistical difference (P < 0.001). Logistic regression model revealed an odds ratio of 1.43 95% confidence interval (CI): 1.06–1.91, P = 0.017 for the possible sarcopenia group and 1.72 (95% CI: 1.04–2.85, P = 0.035) for sarcopenia group when compared with the non‐sarcopenia group. Conclusions Sarcopenia is associated with worse cognitive impairment, which provided new evidence for a strong association that warrants further research into mechanistic insights.