A recent study conducted by Dr. Vadim Axelrod from the Gonda (Goldschmied) Multidisciplinary Brain Research Center at Bar-Ilan University has raised significant concerns regarding the quality of data collected on Amazon Mechanical Turk (MTurk) – a popular platform used for behavioral and psychological research.
MTurk is an online crowdsourcing marketplace where individuals complete small tasks in exchange for payment. For over 15 years, it has been a valuable resource for researchers, despite previous doubts about participant quality. Dr. Axelrod’s research team aimed to assess the current data quality generated by MTurk participants.
The study, which involved over 1,300 participants in main and replication experiments, utilized a simple yet effective method. By repeating identical questionnaire items to measure response consistency, the researchers aimed to gauge the reliability of the participants. Additionally, attentional catch questions were included to test the attentiveness of the respondents.
The results of the study, recently published in the Royal Society Open Science, revealed troubling findings. The majority of participants from MTurk’s general worker pool failed the attention checks and displayed highly inconsistent responses, even among those with a 95% or higher approval rating.
Dr. Axelrod highlighted the significance of these findings, stating that it is challenging to trust data from participants who provide inconsistent responses to basic questions. The study indicated that the participants were not lacking in knowledge but rather in attention to detail, raising concerns about the reliability of their responses to the main questionnaire.
In contrast, Amazon’s elite “Master” workers, selected based on their high performance in previous tasks, consistently produced high-quality data. The researchers recommend using Master workers for future studies due to their experience and reliability.
Dr. Axelrod emphasized the importance of reliable data in empirical science and urged researchers to be cautious when using MTurk’s general worker pool for behavioral research. The study underscores the need for researchers to consider the quality and reliability of their participant pool in order to ensure the validity of their findings.
For more information on the study, titled “Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024,” published in the Royal Society Open Science, please visit the DOI link provided in the article.
This research was conducted by Bar-Ilan University and serves as a reminder of the importance of careful consideration when selecting participants for research studies.
