מיקור המונים במחקר אקדמי: איסוף נתונים ב-Amazon Mechanical Turk

מתוך limudbchevruta

קפיצה אל: ניווט, חיפוש
לתיאור הקבוצה בעברית מיקור המונים במחקר אקדמי 

Crowdsourcing in Academic Research: Data collection via Amazon Mechanical Turk

Crowdsourcing, i.e. sending tasks and assignments for the general public to carry out, mostly over the Internet, has of late been gaining momentum among academic researchers, mainly via the platform of Amazon Mechanical Turk (AMT) The group will bring together doctoral students from different areas of research currently interested in using AMT as a medium for data collection. In the group we will read and discuss important articles dealing with AMT research and attempt to better understand the advantages and disadvantages of this type of academic research. Where possible, the questions raised within the group will be formulated as hypotheses that lend themselves to empirical examination.

Examples of questions we will deal with include: getting to know the AMT population - demographic features, personal characteristics, cultural aspects, and epidemiology of emotional disorders; ensuring data quality; what are the intrinsic differences between experiments run on AMT vs. a university lab (motivation, workload, demographics, environment, etc.); how to deal with experimentation issues that come up on AMT (test-retest, filtering returning participants, increasing participants' trust, etc.); the need for and effectiveness of Instructions Manipulation Checks (IMC) and Attention Check Questions (ACQ); the ratio between payment, data reliability, and the time invested in carrying out the assignment; and ethical questions such as: the level of payment given to test subjects, the date of payment, withholding payment to test subjects who have performed negligently, informing participants about research objectives, sharing information between researchers, and more.


Group Coordinators

Asher Strauss: asher.strauss@gmail.com

Itay Sisso: itaysisso@gmail.com



Group Meetings

First meeting 17 Nov 2014


In our opening meeting Itay presented an overview of Amazon Mechanical Turk (AMT) and methodological issues. The presentation can be found here.  

Second meeting 8 Dec 2014


In the second meeting we discussed the demographics of mTurk participants.


We based our discussion on 4 main papers on this topic:
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples
Ross, J., Irani, L., Silberman, M., Zaldivar, A., & Tomlinson, B. (2010, April). Who are the crowdworkers?: shifting demographics in mechanical turk
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations

Furthermore, as recent developments in Amazon's worker acceptance rules renders the previous demographic data outdated, we decided to include a short 1 minute demographic survey that each one of us will attach to future mTurk experiments and thus collect new data regarding the mTurk workers and their motivations, as well as allow us to detect frauds that answer these questions differently on different occasions. The current version of the Qualtrics questionnaire can be found here.

Third meeting 5 Jan 2014


The third meeting revolved around data quality and reliability issues on mTurk.
Asher presented the following presentation.


We discussed 6 main papers that addressed this issue:
Rouse, S. V. (2015). A reliability analysis of Mechanical Turk data. Computers in Human Behavior
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: consequences and solutions for behavioral researchers
Peer, E., Vosgerau, J., & Acquisti, A. (2013). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2013). Seriousness checks are useful to improve data validity in online research
Antin, J. (2012). Social Desirability Bias and Self-Reports of Motivation : A Study of Amazon Mechanical Turk in the US and India
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power

Fourth meeting 26 Jan 2015


The fourth meeting revolved around payment, bonuses and ethical issues on mTurk.
We based our discussion on an initiative of a Guideline for Academic Requesters

Furthermore, we discussed a mini-conference on mTurk that we intend to arrange towards the end of May 2015. It is planned to last about half a day, and include some new methodological research (by researchers in the Hebrew University and outside of it), as well as an open discussion panel on recent issues. More details to come soon.

Fifth meeting 16 May 2015


The fifth meeting revolved around measuring reaction time (RT) in online experiments, which is a serious challenges for online experimenters. The meeting was conducted by Gidi Aviram a member of Dr. Iftah Yovel lab, who has initial experience with measuring RT online. Gidi introduced the group to the QRTEngine which is an open-source JavaScript engine. It can be embedded in the online survey development environment Qualtrics to develop browser-based online reaction time experiments in an easy way. As preparation for the meeting, we read a paper describing QRTEnging by the developers, published last year in Behavior Research Methods. A copy of the article can be found on the authors website. Following Gidi's introduction we discussed the challenges of measuring RT online, and tried to characterizes RT experiments which are possible to conducted online.

Six meeting 27 April 2015


The sixth meeting revolved around method for implementing online experiments in a web environment. Two members of the group: Itzik Fredkin and Asher Strauss discussed their experience with the group. Itzik introduced us to using Flash and ways of implementing flash based experiments in to the qualtrics environment. Asher discussed his experience with using javaScript. Bothmethods were discussed and compared. One main advantage of flash is that it has a friendlier interface (one may use Adobes programs) and that it is “added on” to qualtrics – which makes it free from qualtrics bugs. On the other hand javaScript is more imbedded in to qualtrics.

Seventh meeting 4 June 2015: Mini-Workshop & Open meeting


The seventh meeting summed up our mutual work this year. We held a open-meeting and mini-work shop with the help of both the Cavruta program and the psychology department. As part of the mini-work shop we had the pleasure to meet Dr. Eyal Pe'er, which is the leading Israeli academic in studying Mechanical Turk (and other platforms) and has vast experience conducting experiments on line. The open meeting attracted many doctoral students from various disciplines as well as academic staff from several departments.

The mini-workshop had to parts:

Part One: Itay Siso and Asher Strauss summed up the groups work, but presenting a bird's-eye view on Mechanical Turk. The abstract for the lecture can be found here. Slides can be found here.

Part Two: Dr. Eyal Pe'er discussed some disadvatages of Mechanical Turk - especially the non-naivety of the Mechanical Turk workers. He presented his study which compared several other platforms, and introduced us to CrowdFlower. It seems the this platform might be a fair alternative for Mechanical Turk, especially for situations where the naivety of Mechanical Turk workers can not be assumed.

The abstract for the lecture can be found here.

Collection of Academic Articles 

A collection of academic articles sorted by tags can be found at our Mendeley group.
כלים אישיים