•Semantic domains in which gesture commonly co-occur with speech were revealed.•Gesture-speech relationship was not significantly affected by speakers’ L1.•Advanced speakers use significantly more gestures as reinforces and integrates.•Less proficient speakers tend to use gestures as complements to speech. This study investigates face-to-face interaction among Taiwanese, Indonesian, and Indian speakers utilizing a multimodal corpus linguistics approach to examine semantic categories of speech that most frequently co-occur with gestures, and whether the gesture-speech relationship is to a certain extent influenced by language/culture backgrounds or English proficiency levels of a speaker. The analysis of the semantic categories of the co-gesture speech demonstrates that speech most commonly co-occurs with gestures in the categories of moving, coming and going, general objects, numbers, location and direction, and time. The findings demonstrate similar preferences of gesture-speech production by speakers despite different cultural and linguistic backgrounds. The gesture-speech relationship was shown to fall into six discrete categories: reinforcing, integrating, supplementary, complementary, contradictory, and others. While results show that the gesture-speech relationship is not significantly influenced by different language backgrounds of a speaker, speakers at a high proficiency level tended to use significantly more gestures that serve reinforcing and integrating functions, whereas less proficient speakers produced more gestures as complements and other gestures that have no obvious relationship to the conceptual content of their accompanying speech.