This study was motivated by a need to understand the extent to which behavioral indicators of engagement from digital log data are associated with various student learning outcomes above and beyond ...self-reported levels of engagement, and whether the strength of these associations vary depending on the type of learning outcome. Student learning was assessed by way of four distinct learning outcomes that varied according to stakes (low-v. high-stakes) and span (one-time v. aggregated). Participants included high school students between 14 and 18 years of age enrolled in an AP Statistics course (N = 320, M age = 16.76 years, SD age = 0.85; 60.2% female) who had consented to use an online assessment system over the course of an academic year that was designed to provide personalized performance reports. While largely uncorrelated with self-report measures, certain process data variables were significantly correlated with learning outcomes. In particular, students’ frequency of score report checking, an indication of feedback-seeking behavior, while uncorrelated with self-reported student engagement, was associated with all learning outcomes. Other behaviors, such as the number of log-in sessions and the duration of sessions, were not. These findings suggest that process data from online assessment systems can help broaden and deepen our understanding of student behavior above and beyond self-report. That said, given that the volume and complexity of process data can make it challenging to mine and interpret, researchers must consider theory when identifying process data variables that are critical to the understanding of constructs of interest.
•We studied the extent process data from an assessment system predicted learning.•We considered learning outcomes differing by low-/high-stakes and one-time/aggregate.•Clicks to results page, an indication of feedback-seeking, predicted all outcomes.•Number of log-in sessions and average session duration did not predict learning.•Process data variables were largely uncorrelated with self-reported engagement.
•A split-sample CFA provided rigorously tested validity evidence for measures used.•Measures were metric invariant for sex, minority status, and parent education.•Teacher support had an effect on ...students' math attitudes.•Investigation of moderation effect in light of measurement invariance.•Benefit of teacher support on math attitudes does not appear to differ based on student background.
We examine the effect of students' perceptions of teacher support on attitudes towards math and whether the association varies between students from historically underrepresented groups in STEM. Participants included high school students enrolled in an AP Statistics course (N = 585, Mage = 16.75 years, SDage = 0.88). Measurement invariance conducted on the basis of student background characteristics (i.e., biological sex, underrepresented race/ethnicity status, and parental educational attainment) provided evidence of metric invariance or greater. Standardized regression coefficients suggested potential differences, such that for students who were female, from an underrepresented racial/ethnicity group in STEM, or had parents with lower levels of education, the effect of teacher support appeared not as strong compared to their counterparts. However, scaled chi-square difference tests comparing nested latent path models did not suggest a moderation effect of these three characteristics. These findings have implications for understanding barriers many students face in receiving the benefits of teacher support.
Response time data from educational and psychological assessments have become more widely available in recent years. Although much research has been done on response time models, there is large ...variability between assessments and the factors that may affect response times. On the person’s side, familiarity with online environments may largely predict response times. On the item’s side, item type and number of words in item stem may account for a substantial variability in response times. In this dissertation, an explanatory framework for response time modeling is proposed. Among the advantages, this explanatory approach provides a more interpretable model for understanding variability in response time due to person or item characteristics, while decreasing the number of parameters to be estimated. Furthermore, including additional information when modeling response times may improve estimation of one’s working speed. Using survey data from a high school sample, we explore also response times, background variables, and process data to illustrate this explanatory framework. Implications and future research directions are also discussed.
We examined whether students were accurate in predicting their test performance in both low-stakes and high-stakes testing contexts. The sample comprised U.S. high school students enrolled in an ...advanced placement (AP) statistics course during the 2017-2018 academic year (N = 209; M
age
= 16.6 years). We found that even two months before taking the AP exam, a high stakes summative assessment, students were moderately accurate in predicting their actual scores (κ
weighted
= .62). When the same variables were entered into models predicting inaccuracy and overconfidence bias, results did not provide evidence that age, gender, parental education, number of mathematics classes previously taken, or course engagement accounted for variation in accuracy. Overconfidence bias differed between students enrolled at different schools. Results indicated that students' predictions of performance were positively associated with performance in both low- and high-stakes testing contexts. The findings shed light on ways to leverage students' self-assessment for learning.
PDF
