To examine the utility of spoken dialog systems (SDSs) for learning and low-stakes assessment, we administered the same role-play task in two different modalities to a group of 47 tertiary-level ...learners of English. Each participant completed the task in an SDS setting with a fully automated agent and engaged in the same task with a human interlocutor in a face-to-face format. Additionally, we gauged students' perceptions of the two delivery formats. Elicited oral performances were examined for linguistic complexity (syntactic complexity, lexical variety, fluency) and pragmatic functions (number and type of requests). Learner performance data across the two delivery modes were comparable although learners spoke slightly longer in the SDS task and used significantly more turns in the face-to-face setting-a finding that may be due to participants deploying more social rapport building moves, clarification requests, and backchanneling. The attitudinal data indicate that, while many learners liked both delivery formats, there was a slight preference for the face-to-face format, mainly due to the presence of body language. Overall, results show that fully automated SDS tasks may constitute a feasible alternative to face-to-face role-plays. Nevertheless, when possible, learners should be given a choice in task format for both learning and assessment.
Often, second/foreign (L2) language learners receive little opportunity to interact orally in the target language. Interactive, conversation-based spoken dialog systems (SDSs) that use automated ...speech recognition and natural language processing have the potential to address this need by engaging learners in meaningful, goal-oriented speaking practice. However, these technology-based learning tools are often developed without input from teaching professionals. As part of a larger development effort, this study examined English as a second language (ESL) teachers' perceptions regarding SDS-based speaking tasks, addressing the following research questions: (a) What do teachers think about the SDS-based tasks? (b) How would they use them in the context of their English instruction? Overall, 16 ESL instructors in an intensive English program in the United States were asked to interact with four SDS-based speaking tasks designed to elicit specific linguistic phenomena (e.g. making requests; wh-questions). The teachers completed a survey after each task to gauge their user experience, level of engagement, and perceptions of the usefulness of the tasks for their teaching contexts. A subgroup of instructors (n = 7) also participated in audio-recorded focus group meetings. Descriptive statistics were calculated for each survey item and the open-ended responses from the surveys and focus groups were analyzed qualitatively for major themes. The findings show that teachers had positive views of the SDS tasks' potential for speaking practice and diagnostic purposes, primarily in a flipped classroom model. Their perceptions seemed to be related to their own user experience and, in particular, to the perceived authenticity of a given task.
In recent years, the number of human-machine interactions has increased considerably. Additionally, we have evidence of linguistic differences between human-machine interactions and human–human ...conversations (e.g., Timpe-Laughlin et al., 2022). Therefore, it is reasonable to revisit theoretical frameworks that conceptualize interactional language use and investigate to what extent they still apply to technology-mediated interactions. As a first attempt at exploring whether pragmatics theories apply to human-machine interaction, we examined how well Kecskés's (2013) socio-cognitive approach (SCA) focusing on asymmetric interactions (e.g., between interlocutors of different language backgrounds) applies to the asymmetry of human-machine interactions.
Using examples from experimental data, we present the nature of common ground between human and machine (spoken dialogue system) interlocutors, focusing on the construction of and reliance on the emergent side of common ground that is informed by the actual situational experience. Like Kecskés, we argue that both egocentrism and cooperation play a role in human-machine interaction. While the former is manifested in approaching the machine interlocutor as if it was human, the latter appears to play a role in common ground seeking and building as well as in recipient design. We demonstrate that Kecskés's SCA is a fitting framework for analyzing human-machine communication contexts.
•Kecskés's socio-cognitive approach (SCA) applies to human-machine interaction.•Both egocentrism and cooperation play a role in human-machine interaction.•Human interlocutors try to find core common ground, and seek to create emergent common ground in interaction with machines.•Emergent common ground is informed by situational experience.•Carefully planned recipient design and salience effect are present in data.
This paper reports findings from a think-aloud study that explored the effectiveness of Words at Work, a learning platform intended to aid adult English-language learners in becoming (more) aware of ...pragmatics in the context of the U.S. workplace. Twenty participants engaged with Words at Work in a computer laboratory. They were asked to (a) think aloud while solving pragmatics tasks (pretest), (b) engage with a Words at Work unit, and (c) complete a parallel form of the pretest pragmatics task, again thinking aloud while solving the tasks (posttest). The audio-recorded verbal reports were transcribed and coded inductively for pragmatic-related episodes (PREs), instances where participants mentioned relevant pragmalinguistic or sociopragmatic aspects. Frequency counts were tabulated for all PREs, and paired-sample t-tests were conducted to investigate differences between pre- and posttest results. Findings revealed statistically significant increases in PREs in the post-think-alouds. In addition to highlighting a change in cognitive state, we observed a key aspect of raising L2 learners’ pragmatic awareness: the need to provide learners with the vocabulary to talk about how they use language. Findings are discussed in terms of their implications for the construct of metapragmatic awareness and effectively designing environments to promote pragmatic awareness.
Accomplishing oral interactive workplace tasks requires various language abilities, including pragmatics. While technology-mediated tasks are thought to offer many possibilities for teaching and ...assessing second language (L2) pragmatics, their effectiveness – especially those facilitated by an AI agent (artificial intelligence agent) – remains to be explored. This study investigated how 47 tertiary-level learners of English as a second language (ESL) performed on an oral interactive task that required them to make requests to their boss in two distinct modalities. Each participant completed the same task with a fully automated AI agent and with a human interlocutor in a face-to-face format. Findings showed that both modalities elicited language use relevant to the pragmatics target. However, fully automated interactions were found to be more transactional, while face-to-face interactions were more functionally oriented (e.g. more frequent/varied supportive moves). Although fully automated interactive tasks may be useful for eliciting requests, replicating human-to-human interactions remains a challenge.
Despite the frequent occurrence of yes/no questions in naturalistic talk-in-interaction, there is widespread concern that yes/no questions only allow predetermined answers and limit L2 learners' ...performance in language assessment contexts. Drawing upon audio-recordings of role-play conversations between an L1 American English speaker and 104 L2 English learners, we use conversation analysis to examine how L2 learners respond to yes/no questions in an oral performance assessment. The analysis was guided by the following research questions: (a) How do L2 English learners respond to a yes/no interrogative in a role-play assessment context? (b) How do L2 English learners' different responses lead to distinct sequences of conversational turns? In our analysis we show that yes/no interrogatives elicit a range of responses (e.g., repair initiation, type-conforming responses, non-type-conforming responses) that result in negotiation of meaning between participants. While turn design and sequence organization features we observed largely align with the existing literature on naturally occurring conversations, we also discuss deviant cases that provide insights into L2 learner discourse. Moreover, we discuss the findings in terms of implications for L2 assessment.
This study reports on the development and evaluation of a prototype for an interactive, self‐access computer application titled Words at Work that was developed by the authors. Words at Work aims to ...raise adult English language learners' (ELLs') pragmatic awareness in the English‐medium workplace. The study focuses in particular on the pedagogic principles of second‐language (L2) instruction underlying the design of the computer‐based learning environment and seeks to explore the perceived usefulness of computer‐assisted language learning (CALL)–mediated L2 pragmatic instruction, responding to this overarching research question: To what extent do learners feel that Words at Work successfully operationalizes the principles of teaching L2 pragmatics as proposed in the research literature? Accordingly, we first review teaching principles for L2 pragmatics instruction and present an overview of existing stand‐alone materials for L2 pragmatics learning and teaching. Then, we outline the development and design features of Words at Work, before presenting the findings of a usability study that was conducted with 19 adult ELLs. We found evidence that users desired immersive learning environments that incorporate authentic audiovisual input, personalized and immediate feedback, and extensive opportunities for interaction and self‐reflection. Implications for the design of computer‐mediated L2 pragmatic learning tools and tasks will be discussed both for instruction and assessment.
Report Number: ETS RR–19‐34
The construct of oral ability is multifaceted, but due to technological and practical constraints, the majority of computer‐delivered speaking assessments are designed to measure only certain aspects ...of this ability. Most notably, interactional competence, which we define as the ability to actively structure dialogic speech in real time, is often not assessed. Innovations in technology, namely, computer‐mediated video, make it possible for test takers in different locations to see and speak with others in real time and may make it achievable for computer‐based tests to assess more aspects of oral communication, including interactional competence. This report gives the findings from a study that explored to what extent computer‐mediated video, namely, Skype, could function in conjunction with a platform designed to present four innovative speaking tasks that could conceivably assess a broad construct of oral ability. The overarching goals of this project were twofold. First, we aimed to determine (a) the degree to which current computer video‐mediated technology can be used effectively to deliver assessments remotely and (b) the extent to which participants felt that four specific tasks could assess speaking ability by means of this technology. The speaking tasks included giving short responses to an interlocutor's questions, summarizing a proposal, defending a position in a group discussion, and giving a prepared presentation and responding to questions from other participants. Two data collections were conducted: one with all 72 participants located in the United States and one with all 74 participants located in China. The findings provide preliminary evidence that the stability of computer video‐mediated technology varies considerably, with technical disruptions being relatively few in the U.S. trial but very frequent in the China context. Moreover, the findings suggest that participants viewed the tasks as generally representing interactive speaking activities that they encounter in the oral language use domain, affording them the opportunity to demonstrate their oral abilities, and that these tasks can be effectively completed in a computer video‐mediated environment when technology cooperates.
Report Number: ETS RR–19‐05
The development of effective second and foreign (L2) language learning materials needs to be grounded in two types of theories: (a) a theory of language and language use and (b) a theory of language ...learning. Both are equally important, insofar as an effective learning environment requires an understanding of the knowledge, skills, and abilities that are being targeted as well as an idea of how learner development progresses. Although the author and colleagues have previously provided a theoretical basis for the construct of L2 pragmatic ability, and thus a theory of language use, the current paper focuses on learner development, particularly on the theoretical orientation towards L2 pragmatic learning. Given this superordinate goal, the paper will briefly review different theoretical paradigms that have been employed in adult L2 pragmatics development research. In the first part of the paper, I will provide an overview of the cognitive, socially oriented, and emergentist perspectives used in research on L2 pragmatic development. Then, based on the different ontological perspectives, I will discuss the affordances and challenges of applying a theoretical perspective to designing L2 pragmatic learning environments, drawing upon the example of the cognitively oriented information processing theory.
PDF
