This exploratory study examines the effect of independent e-book readings compared to listening to an adult reading printed books aloud on young Korean learners’ vocabulary acquisition and parents’ ...perceptions of e-books and printed books as vocabulary learning tools for their children in EFL contexts. In a counterbalanced, within-subject study design, 19 young Korean EFL learners participated in eight 20-25-minute reading sessions (four independent e-book reading and four reading printed book teacher read-aloud) over eight days and completed pre-tests and post-tests that were designed to measure receptive vocabulary learning (word meaning recognition). In addition, 128 Korean parents responded to an online survey. A Wilcoxon signed-rank test revealed both reading methods exerted positive effects on the learners’ incidental L2 vocabulary learning to different extents. Analysis of the survey responses indicated that the parents considered both reading mediums useful for different reasons. They suggested incorporating additional features into e-books to better assist their children in learning L2 vocabulary. Implications regarding the effective use of two reading methods and the design of e-book features, as well as methodological limitations, are discussed. KCI Citation Count: 1
•An investigation of partial design validity for graphic-prompt writing (GPW) tasks.•Commonalities between the constructs of GPW tasks and standardized writing tests.•A significant predictor of GPW ...task scores: academic writing ability.•A weak but significant association between graph familiarity and GPW task scores.•Non-significant effects of computer mode preference on GPW task scores.
This argument-based validation research examines the validity of score interpretations on computer-based graphic-prompt writing (GPW) tasks, centering on the explanation inference. The GPW tasks, designed for English placement testing, measure examinees’ ability to incorporate visual graphic information into their writing. Over 100 ESL students, studying at a public university in the United States, completed GPW tasks and two online questionnaires on graph familiarity (Xi, 2005) and computer mode preference (Lee, 2004) and submitted their standardized English writing test scores. A Pearson product-moment correlation, corrected for attenuation, revealed scores on the GPW tasks and the standardized writing tests had a moderately strong positive relationship (rT1T2 = .50). Multiple linear regression and follow-up correlation analyses showed that GPW task scores were attributed to examinees’ academic writing ability and carried weak, but significant, positive relations to three predictors of graph familiarity (examinees’ experience with and preference for graphs, and graph reading ability). The findings suggest the GPW tasks and the standardized English writing tests assessed different dimensions of the same underlying construct (academic writing ability), and the three predictors of graph familiarity served as sources of construct-irrelevant variance. Theoretical and practical implications of the findings, as well as methodological limitations, are discussed.
This exploratory study compared the effects of two different mediums of reading—interactive e-book reading and paper-based reading—on learners’ reading comprehension. A specific focus was placed on ...literal vs. inferential comprehension. Thirty Korean middle school English language learners were randomly assigned to an interactive e-book reading (treatment) group or a paper-based reading (control) group. A pre-test and five comprehension tests were administered to both groups over a total of six reading sessions. A survey was also conducted to investigate students’ perceptions of how helpful interactive e-book features were in aiding their reading comprehension. Results from the reading comprehension test scores showed no statistical differences between the groups across the five tests. Analysis of literal and inferential questions also showed no significant difference overall. Findings from the survey data along with researcher observation notes suggest that interactive features that are not designed to aid students’ understanding can distract students from the task of reading which may hinder their comprehension. The overall results of this study suggest that it may not be the medium of reading, but how students engage with each medium that can affect their comprehension of text.
This study describes the development process and examines the construct validity of an English placement test of oral communication (EPT OC) developed at a Midwestern university in the United States. ...This test includes a one‐on‐one oral interview and paired discussion task, and test performance is judged on an analytic rating scale. A confirmatory factor analysis conducted on the ratings of 338 students who took the initial fully operational EPT OC revealed the test structure was represented by a correlated four‐factor model with interactional competence, fluency, pronunciation/comprehensibility, and grammar/vocabulary as sub‐constructs, in line with its targeted theoretical framework. Both tasks were effective in measuring the targeted sub‐constructs, but the sub‐constructs were not sufficiently distinct from each other to completely justify a four‐factor model. The findings provide some support for the proposed interpretations of the EPT OC test scores but indicate the need for some modifications to the assessment, such as more thorough rater training and/or revised rating scales to better distinguish the targeted sub‐constructs.
Tóm tắt
Nghiên cứu này mô tả tiến trình phát triển và kiểm tra tính hữu ích của cấu trúc bài thi kĩ năng nói để xếp lớp đầu vào tại một trường đại học ở miền trung Hoa Kỳ. Bài thi này bao gồm một bài phỏng vấn trực tiếp và một bài thảo luận theo cặp. Sinh viên được đánh giá dựa trên công cụ đánh giá bao gồm bốn tiêu chí: năng lực tương tác, độ lưu loát, phát âm/mức độ dễ hiểu, và ngữ pháp/từ vựng, tương ứng với cơ sở lí thuyết được dùng để phát triển bài thi. Bài nghiên cứu này sử dụng phương pháp phân tích nhân tố khẳng định để phân tích dữ liệu điểm thi của 338 sinh viên khi các sinh viên này tham dự kì thi ngay khi kì thi được bắt đầu. Kết quả cho thấy rằng bài phỏng vấn trực tiếp và bài thảo luận theo cặp đều hữu hiệu trong việc đo lường bốn tiêu chí đánh giá ở trên. Tuy nhiên, kết quả từ phân tích mô hình bốn tiêu chí cho thấy rằng bốn tiêu chí đánh giá này không khác biệt hoàn toàn. Kết quả nghiên cứu này hỗ trợ cho việc sử dụng điểm thi của bài thi này cho mục đích mà bài thi được tạo ra. Tuy nhiên, nghiên cứu này chỉ ra sự cần thiết để đào tạo kỹ lưỡng hơn người tham gia đánh giá bài thi và/hoặc sửa đổi thang đánh giá để giúp người đánh giá có thể phân biệt rõ hơn các tiêu chí đánh giá.
The graphic-prompt writing task, where one or multiple visual graphs are provided as source materials, is a type of integrated writing tasks that assesses test takers’ ability to incorporate ...information from the source(s) into their writing. Compared to reading-to-write and reading-listening-to-write tasks, the graphic-prompt writing task is utilized and researched to a limited extent, even though it holds promise for facilitating multimodal literacy (Jewitt, 2005, 2008) and avoiding construct underrepresentation (Messick, 1989) in L2 academic writing assessment (Lim, 2009). Therefore, the present dissertation study aims to validate score interpretations on the graphic-prompt writing task by taking an argument-based approach; this approach to test validation provides systematic guidance on conducting research to collect validity evidence (Kane, 2001, 2006, 2013). To examine the score interpretations, computer-mediated graphic-prompt writing tasks, as well as a five-point scale analytic rating rubric, were developed for English Placement Test purposes under the Evidence-Centered Design Framework (e.g., Mislevy & Haertel, 2006), and an interpretation/use argument (Kane, 2013) for the test was crafted. Various types of evidence were sought to justify the three inferences (evaluation, generalization, and explanation), relevant to the test score interpretations, via a mixed-methods research design. A range of data (graphic-prompt writing test scores, test essays, stimulated recall protocols, standardized English writing test scores, and responses on Graph Familiarity and Test Mode Preference questionnaires) were collected from 101 ESL students studying at a large Midwestern university in the U.S. The three inferences were generally well supported by the evidence yielded from the quantitative analysis, conducted by descriptive statistics, Multi-Faceted Rasch Measurement, Generalizability Theory, disattenuated correlation, and multiple regression, and from a qualitative analysis of the stimulated recalls and discourse features of the test essays. The evaluation inference was upheld by the findings that the raters showed neither central tendency nor halo effects in their ratings, that the rating scale for the graphic-prompt writing test met five of the six criteria of a quality rating scale, and that scores on the graphic-prompt writing test were widely spread across test takers’ different levels of graphic-prompt writing ability. The generalization inference was supported by results showing that variance in test takers’ graphic-prompt writing ability (object of measurement) contributed most to composite and analytic score variances compared with sources of error variance, test score dependability of the three-rater and two-task test design was ≥ .7 for composite and three analytic rating criteria (Graph Description, Organization, and Grammar/Vocabulary) scores, and the required numbers of test tasks and raters needed for Φ ≥ .7 varied depending on score report methods (composite versus analytic) and analytic rating criteria, though using the test design with two tasks and two raters reporting composite scores appeared the most derisible. The explanation inference was backed by findings that writing processes elicited by the graphic-prompt writing test and most discourse features of the test essays were different depending on test takers’ levels of graphic-prompt writing ability, the construct of the graphic-prompt writing test had a moderately strong positive correlation with the construct of standardized English writing tests, and only test takers’ academic writing ability (a construct-relevant factor) significantly contributed to graphic-prompt writing test scores, while their graph familiarity and test mode preference (construct irrelevant factors) did not. The present study provides implications for the construct underlying the graphic-prompt writing test, dependability and validity of the test score interpretations, and test administration designs for the graphic-prompt writing test, as well as validation of the test.
This narrative review aims to delineate current research trends in topical areas, design characteristics, and key findings from empirical studies that investigated the efficacy of Data-Driven ...Learning (DDL) in local educational contexts over the last decade. To this end, we analyzed 36 academic journal papers, as well as graduate theses and dissertations, using PRISMA guidelines. Major findings indicate a predominance of research on the hybrid area, integrating cognitive and affective aspects. Middle-school and university students with varying English proficiency levels were most represented. Mixed-methods design, involving 11-50 participants over 1-8 sessions, was most common, focusing on vocabulary and grammar learning. Pre- and post-tests were most favored and an indirect approach to DDL dominated using a general corpus. In addition, DDL generally benefited English learners across all proficiency levels, with more learning gains observed within groups compared to between groups. Learners recognized DDL’s language learning benefits, showing preferences for group activities. Common issues included challenges in inferring meaning and linguistic patterns, and overly complex example sentences. The review findings provide insights for future research designs and the pedagogical implementation of DDL in local settings.
자동채점 기반 영어 말하기 시험 현황 분석 이진화; 최윤덕; 성민창 ...
영어교육,
06/2023, Volume:
78, Issue:
2
Journal Article
This study investigated the feasibility of adopting an automatic scoring system (ASS) in a domestic English-speaking education context. Scope, test items, assessment criteria, scoring methods, and ...reporting strategies of six overseas English-speaking tests utilizing ASSs were examined. Moreover, a comparative analysis was conducted to identify disparities between ASS-based and non-ASS-based speaking tests. Findings were: 1) some ASS-based tests utilized ASS technology throughout the assessment, while others adopted a hybrid scoring system involving human raters; 2) compared to non-ASS-based tests, ASS-based tests used more test items targeting low-level skills such as sound and forms but fewer test items targeting conversation and discourse level skills; 3) pronunciation, fluency, and vocabulary were widely employed as evaluation criteria with sparse use of organization, content, and task completion in most ASS-based tests; 4) differences were minimal in assessment criteria application and score calculation between ASS-based and non-ASS-based tests; and 5) some ASS-based tests provided criteria-specific results and feedback with total scores and proficiency levels.
