This study examined how human–human collaboration can be achieved through an exchange of verbal information in exchanging information about the referents in a joint action. Knowing other people’s referential intention is fundamental for joint action. Joint action can be achieved verbally by two types of referring expressions, namely, symbolic and deictic referring expressions. Using corpus data, we extracted nouns as typical symbolic references and demonstratives as typical deictic references. We examined whether the word usage of these terms changed when the robot vehicles controlled by the participants repeatedly performed the same collaborative task. We used a novel virtual space for the task because we wanted to control the common ground shared by the participants. The results of the performance indicate that the task completion became more efficient as the participants repeated the task. The referential word use was reduced in both symbolic and deictic references, and this reduction occurred with a grounding process among the collaborators. The study showed that reduction of referential expressions occurs with the grounding process in human–human collaboration and suggests that appropriate collaborative robot systems must deal with the reduction process of referencing in humans.