Abstract Objective The incidence of dementia in the sub-Saharan Africa is rising. However, screening tools for cognitive decline that fits their linguistic and cultural context are lacking. The aim of this study was to determine the accuracy of the Kiswahili version of Montreal Cognitive Assessment (K-MoCA) to detect mild cognitive impairment or dementia among older adults in the rural Tanzania. Methods We recruited 259 community-dwelling older adults in Chamwino district, Tanzania. The concurrent validity and discriminatory power of K-MoCA were examined by comparing its score with IDEA cognitive screening and psychiatrist’s diagnosis using DSM-V, respectively. All the questionnaires were administered in face-to-face interview. Results K-MoCA demonstrated acceptable reliability (Cronbach’s alpha = 0.780). Concurrent validity was evident by its significant correlation with the IDEA screening test (Pearson’s r = 0.651, p < 0.001). Using the psychiatrist’s rating as the reference, the optimal cut-off score for MCI and dementia was 19 and 15, respectively, which yielded the sensitivity of 70% and specificity of 60% for MCI, and sensitivity of 72% and specificity of 60% for dementia. Further analysis indicated that education and age influence performance on K-MoCA. Conclusion Overall, the K-MoCA is a reliable and valid tool for measuring cognitive decline. However, its limited discriminatory power for MCI and dementia may be compromised by the cultural irrelevance of some items.