The aims of this paper are: to provide a comprehensive introduction to eDia, an online diagnostic assessment system; to show how the use of technology can contribute to solve certain crucial problems ...in education by supporting the personalization of learning; and to offer a general reference for further eDia-based studies. The primary function for which the system is designed is to provide regular diagnostic feedback in three main domains of education, reading, mathematics, and science, from the beginning of schooling to the end of the 6 years of primary education. The cognitive foundations of the system, the assessment frameworks, are based on a three-dimensional approach in each domain, distinguishing the psychological (reasoning), the application, and the disciplinary (curricular content) dimensions of learning. The frameworks have been carefully mapped into item banks containing over a 1,000 innovative (multimedia-supported) items in each dimension. The online assessments were piloted, and the system has been operating in experimental mode in over 1,000 schools for several years. This paper outlines the theoretical foundations of the eDia system and summarizes how results from research on the cognitive sciences, learning and instruction, and technology-based assessment have been integrated into a working system designed to assess a large population of students. The paper describes the main functions of eDia and discusses how it supports item writing, constructing tests, online test delivery, automated scoring, data processing, scaling and the provision of feedback both for students and teachers. It shows how diagnostic assessments can be implemented in school practice to facilitate differentiated instruction through regular measurements and to provide instruments for teachers to make formative assessments. Beyond its main function (supporting development toward personalizing education), the eDia platform has been used for assessments in a number of areas from pre-school to higher education both in Hungary and in a number of other countries as well. The paper also reviews results from eDia-based studies and highlights how technology-based assessment extends the possibilities of educational research by making more constructs measurable.
Remote learning during the COVID pandemic has led to short- and long-term consequences for students' learning. So far, data on learning loss in early schooling have been limited. In this paper, we ...evaluate the effect of remote learning on 1st graders' school readiness skills and 2nd–8th graders’ performance in mathematics, reading and science using rich data collected in Hungary before and during the pandemic (n ≈ 55,000). The results show that kindergarten children and 1st–4th-grade students were significantly negatively affected by COVID restrictions compared to their older peers. This difference was extremely large in schools with a high share of disadvantaged students. More specifically, 1st–4th-grade low-SES students made little or no progress while learning from home.
•Remote learning led to short- and long-term consequences for students' learning.•Kindergartners and 1st graders lost the most in mathematics growth.•2nd–4th-grade students' learning loss was the greatest in reading growth.•Long-term learning loss was disproportionately greater among 2nd–4th-grade students.•2nd–4th-grade students came from low-SES schools made little or no progress.
The purpose of this study is to examine cross-national differences in students’ exploration strategies in a computer-simulated CPS (complex problem-solving) environment and to identify similarities ...and qualitative differences in the way Hungarian and Chinese students explore a CPS environment. In a sample of 187 Chinese and 835 Hungarian students (aged 12), we administered problem-solving items via the eDia platform within the MicroDYN approach. After structuring and coding the logfile data, latent class analyses were used to identify students whose problem-solving strategies showed similar patterns. Results indicated that Chinese students employed the most effective and successful exploration strategy, the VOTAT (vary-one-thing-at-a-time) strategy, more frequently and effectively than Hungarian students and that they showed a significantly higher learning effect during testing than their Hungarian peers. These results highlight the possibilities and importance of explicit enhancement of exploration strategies as a tool for learning in a new technological context.
How important are learning strategies or personal attributes for learning outside of domain-specific knowledge or twenty-first-century transversal skills when predicting academic success in higher ...education? To address this question, we conducted a longitudinal study among 1,681 students at one of the leading universities in Hungary. Students took four tests and questionnaires when they entered the university. They measured knowledge from their previous studies (reading and mathematics), twenty-first-century skills like problem-solving and non-cognitive variables in learning strategies, motivation and attitudes. In the next five years, we followed their academic progress with the aim of detecting cognitive and non-cognitive predictors of academic success toward obtaining a degree and developing training programmes for the most important predictors to boost the probability of academic success. The overarching model, which was controlled for students' socio-economic background, determined 18.4% of the variance in later academic success. Domain-specific knowledge from previous studies and learning motivation emerged as the strongest predictor, learning strategies proved to be the most stable of the other mediating factors as an independent predictor, and twenty-first-century transversal skills proved to be unimportant in earning a university degree. Results highlight the importance of transforming higher education so that it is consistent with the expectations of the twenty-first-century labour market, a future-fit education in Hungary.
•Understanding transitions plays an important role in fostering problem-solving.•The level of successful transition varies substantially with problem complexity.•It is most important to master the ...first transition in the problem-solving process.•Four latent profiles are empirically distinguished at the university level.•Number of interactions is a more effective predictive factor than time-on-task.
Problem-solving is an essential transversal and learning skill in the 21st century. To better support it, we need a deeper understanding of its processes, i.e., problem exploration, problem representation and knowledge use. Understanding the mechanism behind the transitions between these processes plays an important role in supporting its development. With a sample of N=1828 first-year university students, this study is the first to measure how students transition between the different problem-solving processes and how this influences their overall problem-solving performance as well as how this is related to their other test-taking behaviours. Results indicate that mastering the first transition – correctly understanding and depicting the problem structure – plays a crucial role in problem-solving. Students who failed to master it regularly failed in the next transition. Problem complexity strongly influenced transitions during the entire problem-solving process. Based on students’ behavioural patterns, we distinguished four qualitatively different latent transition classes: expert transitioners, advanced transitioners, beginner transitioners and non-transitioners. The number of interactions proved to be a more effective profile characteristic, especially on high-complexity problems, than time spent on the problem-solving process. The results of the current study provide important insights into how students’ transition between the different problem-solving processes and how their test-taking behaviour indicates their transition skills.
Innovative assessments of cross-curricular competencies such as complex problem solving (CPS) have currently received considerable attention in large-scale educational studies. This study ...investigated the nature of CPS by applying a state-of-the-art approach to assess CPS in high school. We analyzed whether two processes derived from cognitive psychology, knowledge acquisition and knowledge application, could be measured equally well across grades and how these processes differed between grades. Further, relations between CPS, general mental ability (g), academic achievement, and parental education were explored. Hungarian high school students in Grades 5 to 11 (N = 855) completed MicroDYN, which is a computer-based CPS test, and the Culture Fair Test 20-R as a measure of g. Results based on structural equation models showed that empirical modeling of CPS was in line with theories from cognitive psychology such that the two dimensions identified above were found in all grades, and that there was some development of CPS in school, although the Grade 9 students deviated from the general pattern of development. Finally, path analysis showed that CPS was a relevant predictor of academic achievement over and above g. Overall, results of the current study provide support for an understanding of CPS as a cross-curricular skill that is accessible through computer-based assessment and that yields substantial relations to school performance. Thus, the increasing attention CPS has currently received on an international level seems warranted given its high relevance for educational psychologists.
The present study investigates students' test-taking effort by integrating and comparing traditional self-report questionnaire data and students' test-taking behavior, based on log data analyses. ...Previous studies have shown that different methods often lead to different results. A computer-based measure of complex problem-solving in uncertain situations was used to minimize the influence of factual knowledge on test performance. K-means cluster analysis was used to build groups of students differing in test-taking effort, resulting in 3 distinct groups. The correlation between students' test-taking effort and test performance proved to be weaker based on the self-reported questionnaire data than on their actual test-taking behavior. Both the self-report questionnaire and the log data showed a decrease in test-taking effort during the test. The number of clicks played the largest role in predicting performance. Results suggest that (1) self-report questionnaire data are not consistent with students' actual test-taking behavior and (2) it's not necessary to make the maximum effort to obtain valid test results, but a certain level of effort is needed.
•Item-specific thresholds perform better than constant thresholds.•Number of clicks has a higher correlation with test performance than response time.•Using the number of clicks may increase the validity of response time-based methods.•Self-reported data are not always consistent with students' actual behavior.•Not necessary maximum effort for good test results, just a certain level of effort.
This study focuses on a computer-based training program in inductive reasoning through tasks embedded into mathematical content for 9–11-years-old students (N = 118) and presents the results of the ...evaluation study. The online training consists of 120 playful problems based on Klauer’s “Cognitive training for children” concept and on his theory of inductive reasoning (Klauer, 1989). Both the experimental and the control group in the study consisted of 118 participants. A computer-based inductive reasoning test comprising of 44 multiple-choice items was used in the pre- and posttest to measure the effectiveness of the training (Cronbach α = .91). Both the test and the training tasks were in Arabic context regarding language and used directions and were delivered via the eDia online assessment platform (Csapó & Molnár, 2019). On the posttest, after the six weeks of training the experimental group significantly outperformed the control group by more than one standard deviation. The effect size of the training was in international context high (Cohen d = 1.71). Non-significant variance of the latent slope indicated that there was no significant variability in responding to the intervention program. This study provided evidence that inductive reasoning could be developed even on class level and in a computerised environment very effectively at the age of 9–11 independent of students’ original level of inductive reasoning, school-achievement, gender, and socio-economic status.
Complex problem solving (CPS) is considered an important educational achievement indicator. Previous research has indicated that CPS performance depends to a substantial extent on the way students ...explore problem environments. In this study, we investigated qualitative differences in the way students interact with such environments. In a sample of N = 2226 Hungarian students in Grades 6 to 8, we applied a latent class approach to investigate the use of the principle of isolated variation as an exploration strategy across six CPS tasks that were developed within the MicroDYN approach. Six qualitatively different class profiles emerged: proficient explorers, intermediate explorers, low-performing explorers, rapid learners, emerging explorers, and nonpersisting explorers. We further validated the profiles by comparing the latent classes with regard to students' overall CPS performance and additional indicators of task exploration. In analyzing age-related and gender differences on a cross-sectional level, we found only a small progression toward better performing class profiles from Grade 6 to Grade 8 (e.g., 14.6% of students in Grade 6 were proficient explorers vs. 24.6% in Grade 8; 27.1% of students in Grade 6 were low-performing explorers vs. 25.8% in Grade 8), and there were no substantial gender differences. This study contributes to the understanding of how students interact with complex problems and is the first to address whether variations in these behaviors indicate qualitatively different levels of strategic behavior. We discuss the theoretical underpinnings and potential of identifying class profiles of students' exploration behavior in the field of educational psychology.
•Strategic exploration is key for successful complex problem solving (CPS).•We identify six qualitatively different classes of explorers among students.•Class membership is associated with other CPS performance parameters.•Class membership yields additional information beyond (quantitative) performance scores.•There is significant but small progression with age and no substantial effect of gender on class membership.
PDF
