Understanding users’ needs is crucial to building and maintaining high quality software. Online software user feedback has been shown to contain large amounts of information useful to requirements ...engineering (RE). Previous studies have created machine learning classifiers for parsing this feedback for development insight. While these classifiers report generally good performance when evaluated on a test set, questions remain as to how well they extend to unseen data in various forms. This study evaluates machine learning classifiers’ performance on feedback for two common classification tasks (classifying bug reports and feature requests). Using seven datasets from prior research studies, we investigate the performance of classifiers when evaluated on feedback from different apps than those contained in the training set and when evaluated on completely different datasets (coming from different feedback channels and/or labelled by different researchers). We also measure the difference in performance of using channel-specific metadata as a feature in classification. We find that using metadata as features in classifying bug reports and feature requests does not lead to a statistically significant improvement in the majority of datasets tested. We also demonstrate that classification performance is similar on feedback from unseen apps compared to seen apps in the majority of cases tested. However, the classifiers evaluated do not perform well on unseen datasets. We show that multi-dataset training or zero shot classification approaches can somewhat mitigate this performance decrease. We discuss the implications of these results on developing user feedback classification models to analyse and extract software requirements.
Full text
Available for:
EMUNI, FIS, FZAB, GEOZS, GIS, IJS, IMTLJ, KILJ, KISLJ, MFDPS, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, SBMB, SBNM, UKNU, UL, UM, UPUK, VKSCE, ZAGLJ
AbstractObjectiveTo develop an instrument to evaluate the credibility of anchor based minimal important differences (MIDs) for outcome measures reported by patients, and to assess the reliability of ...the instrument.DesignInstrument development and reliability study.Data sourcesInitial criteria were developed for evaluating the credibility of anchor based MIDs based on a literature review (Medline, Embase, CINAHL, and PsycInfo databases) and the experience of the authors in the methodology for estimation of MIDs. Iterative discussions by the team and pilot testing with experts and potential users facilitated the development of the final instrument.ParticipantsWith the newly developed instrument, pairs of masters, doctoral, or postdoctoral students with a background in health research methodology independently evaluated the credibility of a sample of MID estimates.Main outcome measuresCore credibility criteria applicable to all anchor types, additional criteria for transition rating anchors, and inter-rater reliability coefficients were determined.ResultsThe credibility instrument has five core criteria: the anchor is rated by the patient; the anchor is interpretable and relevant to the patient; the MID estimate is precise; the correlation between the anchor and the outcome measure reported by the patient is satisfactory; and the authors select a threshold on the anchor that reflects a small but important difference. The additional criteria for transition rating anchors are: the time elapsed between baseline and follow-up measurement for estimation of the MID is optimal; and the correlations of the transition rating with the baseline, follow-up, and change score in the patient reported outcome measures are satisfactory. Inter-rater reliability coefficients (ĸ) for the core criteria and for one item from the additional criteria ranged from 0.70 to 0.94. Reporting issues prevented the evaluation of the reliability of the three other additional criteria for the transition rating anchors.ConclusionsResearchers, clinicians, and healthcare policy decision makers can consider using this instrument to evaluate the design, conduct, and analysis of studies estimating anchor based minimal important differences.
Full text
Available for:
BFBNIB, CMK, NMLJ, NUK, PNG, SAZU, UL, UM, UPUK
Although new service development (NSD) studies have contributed to developing systematic approaches to service innovation, their product-oriented and provider-centric perspectives are limited in ...embracing a value cocreation concept. We investigate how Service Design, as a human-centered and creative approach to service innovation, can reframe NSD processes to implement value cocreation. Multiple case studies on Service Design projects indicate that design-centric approaches can contribute to the whole NSD process in a way that connects organizations’ managerial practices to value cocreation, in that (1) contextual and holistic understandings of user experiences can inform value propositions that better fit users’ value-in-use, (2) codesign with creative supporting tools can facilitate value cocreation by helping users better apply their own resources, (3) prototyping can optimize firms’ resource and process configuration to facilitate users’ engagement with the service, (4) aligning system actors to the user experience can organize and mobilize them to better support users’ value creation, and (5) user-centered approaches and methods can help organizational staff build long-term capability for supporting users’ value creation. Based on the link between Service Design, NSD, and value cocreation, we propose a conceptual NSD model, geared toward value cocreation.
Full text
Available for:
NUK, OILJ, SAZU, UKNU, UL, UM, UPUK
The Behaviour Change Technique Taxonomy v1 (BCTTv1) specifies the potentially active content of behaviour change interventions. Evaluation of BCTTv1 showed the need to extend it into a formal ...ontology, improve its labels and definitions, add BCTs and subdivide existing BCTs. We aimed to develop a Behaviour Change Technique Ontology (BCTO) that would meet these needs.
The BCTO was developed by: (1) collating and synthesising feedback from multiple sources; (2) extracting information from published studies and classification systems; (3) multiple iterations of reviewing and refining entities, and their labels, definitions and relationships; (4) refining the ontology via expert stakeholder review of its comprehensiveness and clarity; (5) testing whether researchers could reliably apply the ontology to identify BCTs in intervention reports; and (6) making it available online and creating a computer-readable version.
Initially there were 282 proposed changes to BCTTv1. Following first-round review, 19 BCTs were split into two or more BCTs, 27 new BCTs were added and 26 BCTs were moved into a different group, giving 161 BCTs hierarchically organised into 12 logically defined higher-level groups in up to five hierarchical levels. Following expert stakeholder review, the refined ontology had 247 BCTs hierarchically organised into 20 higher-level groups. Independent annotations of intervention evaluation reports by researchers familiar and unfamiliar with the ontology resulted in good levels of inter-rater reliability (0.82 and 0.79, respectively). Following revision informed by this exercise, 34 BCTs were added, resulting in the first published version of the BCTO containing 281 BCTs organised into 20 higher-level groups over five hierarchical levels.
The BCTO provides a standard terminology and comprehensive classification system for the content of behaviour change interventions that can be reliably used to describe interventions. The development and maintenance of an ontology is an iterative and ongoing process; no ontology is ever 'finished'. The BCTO will continue to evolve and grow (e.g. new BCTs or improved definitions) as a result of user feedback and new available evidence.
Agrarian transformations are imagined globally through digital agriculture. Most ambitious agrarian plans in India also identify the application of digital technologies to drive the next agro‐food ...revolution. However, the concerns around the ‘soft impacts’ of the agri‐digital transition, including socioethical concerns and farmers’ perceptions, have received relatively little attention among innovators and policymakers. Thus, through a combination of qualitative content analysis and topic modelling, this article studies end‐user feedback on smart farming apps in India. Emerging themes from the feedback analysis inform an understanding of the attributes of responsibilisation in policies through which agri‐digitalisation is being planned and introduced. The main findings indicate that agri‐digital innovations need to be inclusive of nonhegemonic vernaculars, responsive to farmers’ data privacy and ownership considerations and focus on producing simple, actionable insights for farmers instead of bombarding them with conflicting ideas. At the policy level, this article argues that the Indian policy landscape needs to be reformed substantially. This further suggests that India may need a new institutional architecture to manage the digital transition in agriculture.
Full text
Available for:
BFBNIB, FZAB, GIS, IJS, KILJ, NLZOH, NUK, OILJ, SAZU, SBCE, SBMB, UL, UM, UPUK
This article presents an analysis of student feedback received via formal biennial survey and informal post-it notes assessment, and advocates for the use of informal assessment methods to supplement ...formal methodologies. A biennial satisfaction survey and an informal post-it notes assessment were employed to collect data from library users and yielded comprehensive, timely, and actionable feedback from the students, faculty, and staff. Feedback received from the two assessment methods allowed the University Library of Columbus (ULC) to gain a greater understanding of user needs and preferences which was used to improve library spaces, resources, and services to increase user satisfaction. While each method has advantages and disadvantages, combining varied assessment methods helped the ULC to gain a more holistic understanding of its users and provided a rich set of actionable data. Furthermore, student feedback is a valuable tool for library advocacy and outreach to both the university community at large and administrators.
Anomalies and faults can be detected, and their causes verified, using both data-driven and knowledge-driven techniques. Data-driven techniques can adapt their internal functioning based on the raw ...input data but fail to explain the manifestation of any detection. Knowledge-driven techniques inherently deliver the cause of the faults that were detected but require too much human effort to set up. In this paper, we introduce FLAGS, the Fused-AI interpretabLe Anomaly Generation System, and combine both techniques in one methodology to overcome their limitations and optimize them based on limited user feedback. Semantic knowledge is incorporated in a machine learning technique to enhance expressivity. At the same time, feedback about the faults and anomalies that occurred is provided as input to increase adaptiveness using semantic rule mining methods. This new methodology is evaluated on a predictive maintenance case for trains. We show that our method reduces their downtime and provides more insight into frequently occurring problems.
•Methodology to combine knowledge- and data-driven anomaly and fault detection.•Deriving interpretable causes using feedback derived from a dashboard application.•Adaptive approach to detect context-aware alerts using minimal human involvement.•Evaluated on a streaming sensor use case in the predictive maintenance domain.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPUK, ZAGLJ, ZRSKP
PDF
