Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study ...participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk-many of which are common to other nonprobability samples but unfamiliar to clinical science researchers-and suggests concrete steps to avoid these issues or minimize their impact.
Amazon Mechanical Turk (MTurk) is widely used by behavioral scientists to recruit research participants. MTurk offers advantages over traditional student subject pools, but it also has important ...limitations. In particular, the MTurk population is small and potentially overused, and some groups of interest to behavioral scientists are underrepresented and difficult to recruit. Here we examined whether online research panels can avoid these limitations. Specifically, we compared sample composition, data quality (measured by effect sizes, internal reliability, and attention checks), and the non-naivete of participants recruited from MTurk and Prime Panels—an aggregate of online research panels. Prime Panels participants were more diverse in age, family composition, religiosity, education, and political attitudes. Prime Panels participants also reported less exposure to classic protocols and produced larger effect sizes, but only after screening out several participants who failed a screening task. We conclude that online research panels offer a unique opportunity for research, yet one with some important trade-offs.
Running Experiments on Amazon Mechanical Turk Ipeirotis, Panagiotis G; Chandler, Jesse; Paolacci, Gabriele
Judgment and Decision Making,
08/2010, Letnik:
5, Številka:
5
Journal Article
Recenzirano
Odprti dostop
Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online ...labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.
Although participants with psychiatric symptoms, specific risk factors, or rare demographic characteristics can be difficult to identify and recruit for participation in research, participants with ...these characteristics are crucial for research in the social, behavioral, and clinical sciences. Online research in general and crowdsourcing software in particular may offer a solution. However, no research to date has examined the utility of crowdsourcing software for conducting research on psychopathology. In the current study, we examined the prevalence of several psychiatric disorders and related problems, as well as the reliability and validity of participant reports on these domains, among users of Amazon’s Mechanical Turk. Findings suggest that crowdsourcing software offers several advantages for clinical research while providing insight into potential problems, such as misrepresentation, that researchers should address when collecting data online.
Although researchers often assume their participants are naive to experimental materials, this is not always the case. We investigated how prior exposure to a task affects subsequent experimental ...results. Participants in this study completed the same set of 12 experimental tasks at two points in time, first as a part of the Many Labs replication project and again a few days, a week, or a month later. Effect sizes were markedly lower in the second wave than in the first. The reduction was most pronounced when participants were assigned to a different condition in the second wave. We discuss the methodological implications of these findings.
Using capture-recapture analysis we estimate the effective size of the active Amazon Mechanical Turk (MTurk) population that a typical laboratory can access to be about 7,300 workers. We also ...estimate that the time taken for half of the workers to leave the MTurk pool and be replaced is about 7 months. Each laboratory has its own population pool which overlaps, often extensively, with the hundreds of other laboratories using MTurk. Our estimate is based on a sample of 114,460 completed sessions from 33,408 unique participants and 689 sessions across seven laboratories in the US, Europe, and Australia from January 2012 to March 2015.
Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this ...market have been assessed on dimensions that are universally relevant to understanding whether, why, and when they should be recruited as research participants. We discuss the characteristics of MTurk as a participant pool for psychology and other social sciences, highlighting the traits of the MTurk samples, why people become MTurk workers and research participants, and how data quality on MTurk compares to that from other pools and depends on controllable and uncontrollable factors.
Web surveys enable efficient data collection, but their usefulness is potentially limited when studying people with disabilities, who often lack Internet access. We test the feasibility of collecting ...web survey data from a sample of state vocational rehabilitation (VR) applicants, inviting nonrespondents to complete a telephone interview instead. People who lacked Internet access were provided with a mobile device and wireless access and were as likely to complete the web surveys as people who already had Internet access. Respondents who elected to complete the survey online versus by telephone differed in level of education and VR experience. These findings suggest that for disability studies, web surveys are an important supplement to, but not a replacement for, traditional data collection efforts.
Despite their prevalence in the marketplace, little empirical attention has been paid to how employee uniforms affect consumer reactions to service experiences. We propose that employee uniforms ...facilitate the shared categorization of employees and their organization in the mind of the customer, which affects many of the inferences that customers draw following service encounters. Study 1 shows that uniforms lead to greater attribution of responsibility to the company for employee behavior, especially following poor service. Studies 2 and 3 show that uniforms also lead to more assimilation of judgments across employees, increasing the impact of one employee's behavior on judgments of other employees of the same organization. Study 3 shows that employee uniforms lead to more extreme judgments of the company following service encounters. It also shows that bad (good) service from a uniformed employee makes competing companies look better (worse), indicating that uniforms can elicit contrast effects across companies. In sum, the mere presence of a uniform on an unsatisfactory service or retail employee can damage judgments of the organization and its employees and improve judgments of rival organizations compared to identical service from a nonuniformed employee. Managers seem unaware of these negative consequences. These same principles are likely to apply to a wide variety of uniformed services, including police, military, firefighters, and health‐care providers.
The online labor market Amazon Mechanical Turk (MTurk) is an increasingly popular source of respondents for social science research. A growing body of research has examined the demographic ...composition of MTurk workers as compared with that of other populations. While these comparisons have revealed the ways in which MTurk workers are and are not representative of the general population, variations among samples drawn from MTurk have received less attention. This article focuses on whether MTurk sample composition varies as a function of time. Specifically, we examine whether demographic characteristics vary by (a) time of day, (b) day of week, and serial position (i.e., earlier or later in data collection), both (c) across the entire data collection and (d) within specific batches. We find that day of week differences are minimal, but that time of day and serial position are associated with small but important variations in demographic composition. This demonstrates that MTurk samples cannot be presumed identical across different studies, potentially affecting reliability, validity, and efforts to reproduce findings.
PDF
