Administering Qualtrics Surveys via Mechanical Turk

Fifty years ago, survey research was prohibitively costly. With new technologies, it is possible to conduct survey research for less than the price of a good meal. Qualtrics and Mechanical Turk are but two utilities that you can harness for cost-effective social science surveying. Here’s the simplest way:

Step 1: Create a Qualtrics account. At Yale, you have three options for Qualtrics accounts: Yale (IT), Law, and SOM. All three are available via the Yale Qualtrics portal. Unfortunately, they do not “talk to each other” well. So, I recommend creating a general Yale account (the top blue link) to facilitate easier collaboration across schools. If you encounter issues, consult the Qualtrics FAQ or a law librarian.

Step 2: Create a survey. After you have created your survey, you’ll need to add some additional elements to it. FIRST, go to Survey Flow, Add a New Element, Select Web Service. Enter this URL: http://reporting.qualtrics.com/projects/randomNumGen.php  While that box is still open, click Add a parameter twice and enter the values: min and 10000 and max 99999. That is, in the first row, write min in the first box and 10000 in the box following the equal sign. This will generate a five-digit code. If you prefer, you could instruct the system to create shorter or longer codes. Click test URL, click the checkbox to the left, click Add Embedded Data. A Set Embedded Data field will appear. Replace the name in the left-most box (random, preceded by ED) with confirmation_code  Then, move the Web Service element to the top of the Survey Flow. Save the Survey Flow. SECOND, click the Library tab (along the main/top row). Click Message Library, click Create a New Message (green button, right-hand side), set the Category to End of Survey Messages, in the Description field write Confirmation Code Message. Paste this into the message box:

Your survey has been submitted successfully.

Below is your MTurk confirmation code:

${e://Field/confirmation_code}

Save the message. THIRD, click the Edit Survey tab (main/top row). Click Survey Options. Scroll down to Survey Termination. Click Custom end of survey message. A Load a Saved Message box will appear. Navigate to My Library and click Confirmation Code Message. Save Changes. FOURTH, if you don’t want sophisticated MTurk users to skip through the survey to the end to get the code, use Validation Options: Force Response for each question (along the right side, under Item Type, etc.).

Thanks to the adventurous PhD Psychology Student (his handle) for excellent MTurk advice!

For a more complex/advanced method of connecting Qualtrics and MTurk, see Daniel Zhou’s blog.

Step 3: Do NOT distribute your survey. Instead, consider whether you need to complete the human subjects review/Institutional review board (IRB) process. You should do so if you ever hope to publish the results of your research (e.g., in a law review or social science journal). Most credible outlets won’t publish human subjects research that has not received IRB approval, and conducting human subjects research without IRB approval can be viewed by academic hiring committees as a form of academic misconduct. For more on IRBs generally and links to key Yale University IRB pages (note: the law school does not maintain a separate IRB), read this library blog post: http://library.law.yale.edu/news/human-subjects-research-review-basics-irb

Step 4: Create a Mechanical Turk account (i.e., via the “Get Results…” section of the landing page), and then create a Survey Link project: Click the Create tab and Select Survey Link. Enter in descriptions and details for your project, such as how much you want to pay each participant, or “worker” in MTurk parlance. Click Save and then Click the Design Layout button. Enter in a description of the task. Keep it short, but also consider pasting your informed consent language several lines below the description of the task. Keep the code box. Paste in your Qualtrics survey link: To get your survey link, toggle back into Qualtrics, click into the survey, and click the Launch Survey (globe) button near the top. You might need to activate the survey; you will receive an URL to paste into MTurk. Now that you’re back in MTurk, finish the survey and click the New Batch button. Preview and confirm your batch and click Publish HITs. This launches your project in MTurk.

Step 5: Download your results in CSV format directly from MTurk. It is possible to import results back into Qualtrics, but the within-Qualtrics statistical analysis functionality is narrow, so this option is of limited utility.

For more information, consult the titles and links below:

1. Book: Internet, mail, and mixed-mode surveys: the tailored design method

2. APA dictionary of statistics and research methods (2014)

3. Sage handbook of criminological research methods

4. Qualtrics quick-start guide by Jonathan Snow

5. Daniel Zhou’s Qualtrics/MTurk blog

6. MTurk and Qualtrics tricks blog

7. MTurk experimental design and sampling issues, paper by Paolacci et al.

8. Participant screening during multiple survey administration, paper by Peer et al. 

Image: Survey #4, painting by Barbara Kruger. Courtesy of Yale Digital Image Collection.