We aimed to investigate the association between oral health and cognitive function in a sample of elderly people from a Chinese rural community. The cross-sectional cognitive function of 677 individuals were assessed by Mini-Mental State Examination (MMSE) and Montreal Cognitive Assessment (MoCA). A comprehensive profile of the oral health status was evaluated by questionnaire and clinical examination. Multiple covariates-adjusted regression models demonstrated decayed teeth (DT) and decayed/missing/filled teeth (DMFT) were negatively associated with MoCA score (all p < 0.05). Calculus index (CI) and clinical attachment loss (CAL) were significantly associated with the lower MoCA, short-term memory and executive function score, respectively (all p < 0.05). Additionally, participants with missing teeth unrestored tend to get lower MMSE and MoCA scores (p < 0.05). The results also showed that increased DT and CI were modestly associated with higher odds of cognitive impairment (p < 0.05). There is an association between oral health and global cognition. Poor periodontal status was strongly associated with worse global cognition performance, especially in the short-term memory and executive domain for the aging population. Our findings provided insights into the oral and neurological public health concerns and bases for the research of solutions to dementia prevention.