Background Community health workers (CHWs) have been identified as a critical bridge to reaching many communities with essential health services based on their social and geographical proximity to community residents. However, various challenges limit their performance, especially in low-and middle-income countries. With the view to guiding global and local stakeholders on how best to support CHWs, this study explored common challenges of different CHW cadres in various contexts. Methods We conducted 36 focus group discussions and 131 key informant interviews in Bangladesh, India, Kenya, Malawi, and Nigeria. The study covered 10 CHW cadres grouped into Level 1 and Level 2 health paraprofessionals based on education and training duration, with the latter having a longer engagement. Data were analysed using thematic analysis. Results We identified three critical challenges of CHWs. First, inadequate knowledge affected service delivery and raised questions about the quality of CHW services. CHWs’ insufficient knowledge was partly explained by inadequate training opportunities and the inability to apply new knowledge due to equipment unavailability. Second, their capacity for service coverage was limited by a low level of infrastructural support, including lack of accommodation for Level 2 paraprofessional CHWs, inadequate supplies, and lack of transportation facilities to convey women in labour. Third, the social dimension relating to the acceptance of CHWs’ services was not guaranteed due to local socio-cultural beliefs, CHW demographic characteristics such as sex, and time conflict between CHWs’ health activities and community members’ daily routines. Conclusion To optimise the performance of CHWs in LMICs, pertinent stakeholders, including from the public and third sectors, require a holistic approach that addresses health system challenges relating to training and structural support while meaningfully engaging the community to implement social interventions that enhance acceptance of CHWs and their services.