Social media metrics have a genuine networked nature, reflecting the networking characteristics of the social media platform from where they are derived. This networked nature has been relatively ...less explored in the literature on altmetrics, although new network‐level approaches are starting to appear. A general conceptualization of the role of social media networks in science communication, and particularly of social media as a specific type of interface between science and society, is still missing. The aim of this paper is to provide a conceptual framework for appraising interactions between science and society in multiple directions, in what we call heterogeneous couplings. Heterogeneous couplings are conceptualized as the co‐occurrence of science and non‐science objects, actors, and interactions in online media environments. This conceptualization provides a common framework to study the interactions between science and non‐science actors as captured via online and social media platforms. The conceptualization of heterogeneous couplings opens wider opportunities for the development of network applications and analyses of the interactions between societal and scholarly entities in social media environments, paving the way toward more advanced forms of altmetrics, social (media) studies of science, and the conceptualization and operationalization of more advanced science‐society studies.
This article discusses a project under development called “Inventing Indicators of Interdisciplinarity,” as an example of work in methodology development that combines quantitative methods with ...interpretative approaches in social and cultural research. Key to our project is the idea that Science and Technology Indicators not only have representative value, enabling empirical insight into fields of research and innovation but simultaneously have organizing capacity, as their deployment enables the curation of communities of interpretation. We begin with a discussion of concepts and methods for the analysis of interdisciplinarity in Science and Technology Studies (STS) and scientometrics, stressing that both fields recognize that interdisciplinarity is contested. To make possible a constructive exploration of interdisciplinarity as a contested—and transformative—phenomenon, we sketch out a methodological framework for the development and deployment of “engaging indicators.” We characterize this methodology of
as participatory, abductive, interactive, and informed by design, and emphasize that the method is inherently combinatory, as it brings together approaches from scientometrics, STS, and humanities research. In a final section, we test the potential of our approach in a pilot study of interdisciplinarity in AI, and offer reflections on digital mapping as a pathway towards
interdisciplinarity.
This review of the international literature on evaluation systems, evaluation practices, and metrics (mis)uses was written as part of a larger review commissioned by the Higher Education Funding ...Council for England (HEFCE) to inform their independent assessment of the role of metrics in research evaluation (2014–5). The literature on evaluation systems, practices, and effects of indicator uses is extremely heterogeneous: it comprises hundreds of sources published in different media, spread over disciplines, and with considerable variation in the nature of the evidence. A condensation of the state-of-the-art in relevant research is therefore highly timely. Our review presents the main strands in the literature, with a focus on empirical materials about possible effects of evaluation exercises, ‘gaming’ of indicators, and strategic responses by scientific communities and others to requirements in research assessments. In order to increase visibility and availability, an adapted and updated review is presented here as a stand-alone—after authorization by HEFCE.
Portfolios of Worth Rushforth, Alexander; Franssen, Thomas; de Rijcke, Sarah
Science, technology & human values/Science, technology, & human values,
03/2019, Letnik:
44, Številka:
2
Journal Article
Recenzirano
Odprti dostop
How are “interesting” research problems identified and made durable by academic researchers, particularly in situations defined by multiple evaluation principles? Building on two case studies of ...research groups working on rare diseases in academic biomedicine, we explore how group leaders arrange their groups to encompass research problems that latch onto distinct evaluation principles by dividing and combining work into “basic-oriented” and “clinical-oriented” spheres of inquiry. Following recent developments in the sociology of (e)valuation comparing academics to capitalist entrepreneurs in pursuit of varying kinds of worth, we argue that the metaphor of the portfolio is helpful in analyzing how group leaders manage these different research lines as “alternative investment options” from which they were variously hoping to capitalize. We argue portfolio development is a useful concept for exploring how group leaders fashion “entrepreneurial” practices to manage and exploit tensions between multiple matrices of (e)valuation and conclude with suggestions for how this vocabulary can further extend analysis of epistemic capitalism within science and technology studies.
Filling in the gaps Kaltenbrunner, Wolfgang; de Rijcke, Sarah
Social studies of science,
12/2019, Letnik:
49, Številka:
6
Journal Article
Recenzirano
Odprti dostop
In this article, we study the use of curricula vitae (CV) for competitive funding decisions in science. The typically sober administrative style of academic résumés evokes the impression of ...straightforwardly conveyed, objective evidence on which to base comparisons of past achievements and future potentials. We instead conceptualize the evaluation of biographical evidence as a generative interplay between an historically grown, administrative infrastructure (the CV), and a situated evaluative practice in which the representational function of that infrastructure is itself interpreted and established. The use of CVs in peer review can be seen as a doubly comparative practice, where referees compare not only applicants (among each other or to an imagined ideal of excellence), but also their own experience-based understanding of practice and the conceptual assumptions that underpin CV categories. Empirically, we add to existing literature on peer review by drawing attention to self-correcting mechanisms in the reproduction of the scientific workforce. Conceptually, we distinguish three modalities of how the doubly comparative use of CVs can shape the assessment of applicants: calibration, branching out, and repair. The outcome of this reflexive work should not be seen as predetermined by situational pressures. In fact, bibliographic categories such as authorship of publications or performance metrics may themselves come to be problematized and reshaped in the process.
The rise of new modes evaluating academic work has substantially changed institutions and cultures of knowledge production. This has been reflected and criticized in the literature in STS and beyond. ...For STS scholars, these debates (should) however have an even more specific dimension. Many of us are experts on aspects of these changes. But at the same time, we too are part of the processes we are analyzing, and often criticizing. To put it slightly provocatively, often we cannot avoid playing the very same game that we scrutinize. This creates tensions that many of us reflect on, and it certainly has created many implicit and explicit normative stances on how to deal with them. Yet it seems that so far there has been little room in our field to reflect on and exchange this particular kind of experience-based knowledge. There are many different ways to engage with the dynamics of evaluation, measurement and competition in contemporary academia, or to play what we refer to colloquially here as the “indicator game.” With this debate, we would like to give room to the expression and discussion of some of these ways. This text is the introduction and prompt to an experimental debate. We discuss the state of the academic discussion on the impact of indicator-based evaluation on academic organization, epistemic work and identities. We use insights from these debates to raise questions for how STS and STSers themselves deal with the indicator game. In conclusion, we summarize our contributors’ arguments and propose the concept of “evaluative inquiry” as a new way of representing the quality of STS work in evaluative contexts.
The range and types of performance metrics has recently proliferated in academic settings, with bibliometric indicators being particularly visible examples. One field that has traditionally been ...hospitable towards such indicators is biomedicine. Here the relative merits of bibliometrics are widely discussed, with debates often portraying them as heroes or villains. Despite a plethora of controversies, one of the most widely used indicators in this field is said to be the Journal Impact Factor (JIF). In this article we argue that much of the current debates around researchers' uses of the JIF in biomedicine can be classed as 'folk theories': explanatory accounts told among a community that seldom (if ever) get systematically checked. Such accounts rarely disclose how knowledge production itself becomes more-or-less consolidated around the JIF. Using ethnographic materials from different research sites in Dutch University Medical Centers, this article sheds new empirical and theoretical light on how performance metrics variously shape biomedical research on the ' shop floor.' Our detailed analysis underscores a need for further research into the constitutive effects of evaluative metrics.
Global university rankings have become increasingly important 'calculative devices' for assessing the 'quality' of higher education and research. Their ability to make characteristics of universities ...'calculable' is here exemplified by the first proper university ranking ever, produced as early as 1910 by the American psychologist James McKeen Cattell. Our paper links the epistemological rationales behind the construction of this ranking to the sociopolitical context in which Cattell operated: an era in which psychology became institutionalized against the backdrop of the eugenics movement, and in which statistics of science became used to counter a perceived decline in 'great men.' Over time, however, the 'eminent man,' shaped foremost by heredity and upbringing, came to be replaced by the excellent university as the emblematic symbol of scientific and intellectual strength. We also show that Cattell's ranking was generative of new forms of the social, traces of which can still be found today in the enactment of 'excellence' in global university rankings. (HRK / Abstract übernommen).
PDF
