Even with obvious deficiencies, large prompt-commanded multimodal models are proving to be flexible cognitive tools representing an unprecedented generality. But the directness, diversity, and degree ...of user interaction create a distinctive “human-centred generality” (HCG), rather than a fully autonomous one. HCG implies that —for a specific user— a system is only as general as it is effective for the user’s relevant tasks and their prevalent ways of prompting. A human-centred evaluation of general-purpose AI systems therefore needs to reflect the personal nature of interaction, tasks and cognition. We argue that the best way to understand these systems is as highly-coupled cognitive extenders, and to analyse the bidirectional cognitive adaptations between them and humans. In this paper, we give a formulation of HCG, as well as a high-level overview of the elements and trade-offs involved in the prompting process. We end the paper by outlining some essential research questions and suggestions for improving evaluation practices, which we envision as characteristic for the evaluation of general artificial intelligence in the future.
This paper appears in the AI & Society track.
Planning is an essential cognitive process of executive functions and is considered as one of the most important brain functions. Planning has been extensively studied in the field of ...neuropsychology, but there is a lack of computerized assessment tools of planning ability that are easily accessible to researchers and clinicians. The present study aims to validate a newly developed online spatial network test that is designed for both clinical and nonclinical populations. Exploratory factor analysis revealed a unidimensional factorial structure with moderate to high internal consistency in the test. Two-parameter logistic item response theory analysis showed acceptable item and model fit and no violation of the local independence assumption. The overall success rate exhibited a positive correlation with the performance of planning before attempting to solve the items. After correcting for attenuation, moderate to high correlations were found between the spatial network test and the International Cognitive Ability Resource 16 short form cognitive ability test and the automated perceptual maze test, respectively, demonstrating convergent, discriminant, and criterion validity. Future directions and potential applications of the spatial network test as an assessment tool to measure planning for researchers and clinicians are discussed in the end.
This study investigates the item properties of a newly developed Automatic Number Series Item Generator (ANSIG). The foundation of the ANSIG is based on five hypothesised cognitive operators. ...Thirteen item models were developed using the numGen R package and eleven were evaluated in this study. The 16-item ICAR (International Cognitive Ability Resource
) short form ability test was used to evaluate construct validity. The Rasch Model and two Linear Logistic Test Model(s) (LLTM) were employed to estimate and predict the item parameters. Results indicate that a single factor determines the performance on tests composed of items generated by the ANSIG. Under the LLTM approach, all the cognitive operators were significant predictors of item difficulty. Moderate to high correlations were evident between the number series items and the ICAR test scores, with high correlation found for the ICAR Letter-Numeric-Series type items, suggesting adequate nomothetic span. Extended cognitive research is, nevertheless, essential for the automatic generation of an item pool with predictable psychometric properties.
As COVID-19 vaccinations became available and were proven effective in preventing serious infection, uptake amongst individuals varied, including in medically vulnerable populations. This ...cross-sectional multi-site study examined vaccine uptake, hesitancy, and explanatory factors amongst people with serious and/or chronic health conditions, including the impact of underlying disease on attitudes to vaccination. A 42-item survey was distributed to people with cancer, diabetes, or multiple sclerosis across ten Australian health services from 30 June to 5 October 2021. The survey evaluated sociodemographic and disease-related characteristics and incorporated three validated scales measuring vaccine hesitancy and vaccine-related beliefs generally and specific to their disease: the Oxford COVID-19 Vaccine Hesitancy Scale, the Oxford COVID-19 Vaccine Confidence and Complacency Scale and the Disease Influenced Vaccine Acceptance Scale-Six. Among 4683 participants (2548 54.4% female, 2108 45.0% male, 27 0.6% other; mean SD age, 60.6 13.3 years; 3560 76.0% cancer, 842 18.0% diabetes, and 281 6.0% multiple sclerosis), 3813 (81.5%) self-reported having at least one COVID-19 vaccine. Unvaccinated status was associated with younger age, female sex, lower education and income, English as a second language, and residence in regional areas. Unvaccinated participants were more likely to report greater vaccine hesitancy and more negative perceptions toward vaccines. Disease-related vaccine concerns were associated with unvaccinated status and hesitancy, including greater complacency about COVID-19 infection, and concerns relating to vaccine efficacy and impact on their disease and/or treatment. This highlights the need to develop targeted strategies and education about COVID-19 vaccination to support medically vulnerable populations and health professionals.
Background: Vaccination is the cornerstone of the global public health response to the COVID-19 pandemic. Excess morbidity and mortality of COVID-19 infection is seen in people with cancer. COVID-19 ...vaccine hesitancy has been observed in this medically vulnerable population, although associated attitudes and beliefs remain poorly understood. Methods: An online cross-sectional survey of people with solid organ cancers was conducted through nine health services across Australia. Demographics, cancer-related characteristics and vaccine uptake were collected. Perceptions and beliefs regarding COVID-19 vaccination were assessed using the Oxford COVID-19 Vaccine Hesitancy Scale, the Oxford COVID-19 Vaccine Confidence and Complacency Scale and the Disease Influenced Vaccine Acceptance Scale-6. Results: Between June and October 2021, 2691 people with solid organ cancers completed the survey. The median age was 62.5 years (SD = 11.8; range 19–95), 40.9% were male, 71.3% lived in metropolitan areas and 90.3% spoke English as their first language. The commonest cancer diagnoses were breast (36.6%), genitourinary (18.6%) and gastrointestinal (18.3%); 59.2% had localized disease and 56.0% were receiving anti-cancer therapy. Most participants (79.7%) had at least one COVID-19 vaccine dose. Vaccine uptake was higher in people who were older, male, metropolitan, spoke English as a first language and had a cancer diagnosis for more than six months. Vaccine hesitancy was higher in people who were younger, female, spoke English as a non-dominant language and lived in a regional location, and lower in people with genitourinary cancer. Vaccinated respondents were more concerned about being infected with COVID-19 and less concerned about vaccine safety and efficacy. Conclusions: People with cancer have concerns about acquiring COVID-19, which they balance against vaccine-related concerns about the potential impact on their disease progress and/or treatment. Detailed exploration of concerns in cancer patients provides valuable insights, both for discussions with individual patients and public health messaging for this vulnerable population.
The Elithorn perceptual maze test is widely used in clinical research and practice. However, there is little evidence of its psychometric properties, and its application is limited by the technical ...difficulty of developing more mazes. The current research aims to adopt a rigorous approach to evaluate 18 mazes that were automatically generated by a novel R software package. Various item response theory models were employed to examine the difficulty parameters. The findings suggested that the data best fitted the Rasch model. The linear logistic test model revealed meaningful contribution to the sources of maze difficulty. Additionally, the linear logistic test model plus error was considered the most parsimonious model. The Automatic Perceptual Maze Test was moderately correlated with a nonverbal intelligence test. By introducing more mazes to provide adequate information on participants’ ability at all levels, the Automatic Perceptual Maze Test promises future clinical and research utility for the study of cognitive performance.
Patient-reported assessments are transforming many facets of health care, but there is scope to modernize their delivery. Contemporary assessment techniques like computerized adaptive testing (CAT) ...and machine learning can be applied to patient-reported assessments to reduce burden on both patients and health care professionals; improve test accuracy; and provide individualized, actionable feedback. The Concerto platform is a highly adaptable, secure, and easy-to-use console that can harness the power of CAT and machine learning for developing and administering advanced patient-reported assessments. This paper introduces readers to contemporary assessment techniques and the Concerto platform. It reviews advances in the field of patient-reported assessment that have been driven by the Concerto platform and explains how to create an advanced, adaptive assessment, for free, with minimal prior experience with CAT or programming.
Abstract
Background and Hypothesis
Grandiose delusions may entail difficult responsibilities and detrimental actions for patients. Recognition of these consequences by patients may provide an avenue ...for engagement in treatment. Furthermore, when patients carry out actions within the delusional system (“immersion behaviors”) or spend considerable time thinking about their grandiose beliefs this may contribute to the persistence of the grandiosity and further harmful consequences. We, therefore, investigated grandiose-related subjective harm, immersion behaviors, and perseverative thinking.
Study Design
A cross-sectional study with 798 patients with psychosis (375 of whom had grandiose delusions) and 4518 nonclinical adults. Factor analyses using data from participants scoring highly on grandiosity were used to form 3 scales: subjective harm from exceptional experiences questionnaire; immersion behaviors questionnaire; and thinking about exceptional experiences questionnaire. Associations with grandiosity were tested using structural equation modeling.
Study Results
A total of 268 (77.9%) patients with grandiose delusions identified grandiose-related harms in the past 6 months and 199 (55.1%) wanted help. Immersion behaviors and perseverative thinking were highly prevalent, and explained 39.5% and 20.4% of the variance in grandiosity, respectively. Immersion behaviors and perseverative thinking were significantly associated with subjective harm, even when severity of grandiosity was controlled. Requests for help were associated with higher levels of subjective harm, use of immersion behaviors, and perseverative thinking but not severity of grandiosity.
Conclusions
Acting on grandiose delusions, including harmful behaviors and excessive thinking about grandiose delusions, may be routes for clinicians to engage patients in treatment. This could be a starting point for targeted psychological interventions for grandiose delusions.
•The Knowledge of Genome Sequencing (KOGS) scale is a robust new 9-item measure.•The KOGS was developed using a rigorous psychometric development approach.•The KOGS can be administered to patients, ...research participants and public.•The KOGS can also be administered to non-genetics clinicians.•The KOGS may be useful to researchers and clinicians using genome sequencing.
Whole-genome sequencing is being implemented in research and clinical care, yet tools to assess patients’ knowledge are lacking. Our aim was to develop a robust measure of whole-genome sequencing knowledge suitable for patients and other stakeholders including research participants, public, students, and healthcare professionals.
An initial set of 17 items was developed via an iterative process including literature review, expert consultation, focus groups, and cognitive interviews with patients, and then administered to 243 individuals. We used exploratory factor analysis and item-response theory to confirm the psychometric suitability of the candidate items for assessing whole-genome sequencing knowledge.
There was a strong main component after removing 5 items with low factor loadings. Item and scale homogeneity was achieved using Mokken scale analysis. Three further items were removed because they were misfits, inverse duplicates or resulted in local dependency. The remaining nine items fitted the two-parameter logistic IRT model which achieved excellent fit to the observed data. Cronbach’s alpha was 0.79 indicating acceptable reliability.
The KOGS, developed using a rigorous psychometric approach, is a brief and reliable tool.
The KOGS may prove useful for researchers and healthcare professionals using whole-genome sequencing with patients and other stakeholders.
PDF
