While many people use our crowd to run surveys, the answers are typically subjective and thus bypass all of our quality control mechanisms, such as test questions. Specifically, tasks like demographic or psychological surveys are hard to quality control as workers can easily share codes with other workers. A unique token in your survey is not adequate control to limit workers from malicious work and you will have to pay all contributors who complete the task.
If you choose to run external surveys on our platform, we recommend launching them to Level 3 crowds only and paying a small amount to complete the survey and using the manual bonusing function to bonus a large amount if the survey is correctly completed. Be sure to ask for the contributor ID on the CF and external survey to keep track of who to bonus. We cannot guarantee the quality of answers and can not refund any part of the job.
As the platform is not designed for these kinds of surveys the support team is not equipped to help with any implementation details or job design. We also can’t be held responsible for any discrepancies you may see in contributor numbers in your job and in your external survey.
The following tutorial will walk you through the process of creating and launching an external survey job with services like Qualtrics and Survey Monkey on the CrowdFlower Platform. There are also alternate steps for building and running a survey job created on the CrowdFlower platform.
1. Create a Job
- Once signed in to CrowdFlower, create a new job from scratch.
- On the resulting page, you will have the option to upload data, but since you are conducting a survey, you will not need data. Under the data upload box, click Create a Survey.
- Once redirected to the Design page, you will be tasked with creating instructions for your job’s contributors (survey participants). Be sure to provide a clear and concise title that describes your survey well. Avoid displaying payment amounts, as payment often varies across multiple channels.
Fig. 1: Entering Title & Instructions in the Graphical Editor
2. Add a link to the Questionnaire
Now it's time to add a link to your external questionnaire.
- Scroll to the bottom of the Design page and click Code Editor.
- Then, insert the code below where you would like your link to appear:
<a class="clicked validates-clicked" href="http://YOUR_SURVEY.COM/SURVEY.HTML" target="_blank">Click Here to visit the survey</a>
Note: Make sure to wrap your hyperlink inside the CrowdFlower click validator, as is, included in the code above. This ensures your users have clicked on your link before submitting the page. Also, include the target attribute with a value of _blank to ensure the link opens in a new tab.
Alternate step if you are creating a survey on the CrowdFlower platform:
Select ‘Drop your data file here or browse’ on the Data Page and upload a CSV with one row of data (not including headers). It doesn’t matter what’s in the row, just as long as the platform recognizes there is data in the job.
3. Add a text field on your Questionnaire asking contributors for their IDs
In order to ensure each contributor has finished your survey, provide a text field where users must submit their contributor ID upon completing the survey. This will ensure that your contributors have finished your survey honestly. You may also check the CrowdFlower results to guarantee each Contributor ID is on both the CrowdFlower side and your survey side. If they do not match up, please contact email@example.com.
4. Add a text field to validate that users have finished the page
In order to ensure each contributor has finished the page, provide a text field where users must submit some code upon completing the survey.
- Add the code you would like users to submit on the last page of your survey. Ensure that your validation code uses letters and numbers. Do not use punctuation in your code.
- Add your validating text field using the following CML (CrowdFlower Markup Language) at the end of the form:
<cml:text label="Text" data-validates-regex="YOUR CODE HERE" validates="required ss-required regex" data-validates-regex-message="Please copy and paste the code here that can be found at the end of the Survey" default="Enter here..." instructions="Enter Survey code in this field after completing" />
Note: Be sure to replace the value “YOUR CODE HERE” with the code you defined in step 1.
Fig. 2: Creating a Validation Label in the Code Editor
You may preview the job by clicking on the "Preview" button located in the right side menu. This preview gives you the ability to see the page as it will appear to the contributors.
If you are creating a survey on the CrowdFlower platform, you can skip this step.
5. Adjust the Job Settings
After your survey questionnaire has been constructed, you will need to make adjustments to your Settings. To do this, on the right-side navigation pane, click Settings.
Define the following settings:
Level Settings (Settings > Contributors)
- Decide if you would like to emphasize speed or performance for your survey. 'Level 3' results will be collected more slowly but have the most trusted workers. Accuracy will decrease with 'Level 2' and 'Level 1', but each level lower will include more qualified contributors and throughput will be faster.
Channels (Settings > Contributors)
- Enable the External Contributors to send the survey to the crowd.
- Check "Internal Contributors" if you would like people within your organization to participate in the survey or if you would like to test the user experience of your job.
Geography/Language (Settings > Contributors > Geography/Language)
- Select which countries you would like to send the survey to. Only people from the countries you select will be able to participate in your survey. Location is based on IP address.
Max Judgments per Contributor (Settings > Quality Control > Quality Control Settings)
- Set "Max Judgments per Contributor" to 1 so that a contributor can only answer your survey once.
Minimum Time Per Page (Settings > Quality Control > Quality Control Settings)
- Set "Minimum time it should take a contributor to complete a page of work" to something appropriate so that a contributor is not paid for spending too little time on your survey.
Judgment Details (Launch)
- Set "Judgments per Row" to the number of contributors you would like to take your survey.
Payment Details (Settings > Pay OR Launch)
- Set the payment amount for your survey. In finding the right price for your job, it is best to start from your budget and work backwards. Please also be aware that channels do not always pay in US cash & channels take a cut of the cost so posting your price is not advised.
Set the number of rows displayed on each page (Settings > Pay)
- Set "Rows per Page" to 1
6. Preview the Job
Click on the 'Preview' link on the right of any page:
If the job looks acceptable move on to step 7.
7. Launch the Job
- To get your job up and running, click the Launch button on the top navigation pane.
- Check your payment and see that you are only launching 1 row.
- Click Launch Job Now in the top right corner.
Fig. 3: Launch Page
7. Monitor Your Job
You can keep an eye on incoming judgments and the progress of your job on the Monitor page. This page is accessible by clicking the Monitor button located on the top navigation pane.
Fig. 4: Monitor your Job
8. Download Results
As soon as the job as the job has finished collecting judgments, reports generate automatically that show data on contributors, such as location, channel, time, and IP address. You can access these reports by clicking on the Results button located on the top navigation pane.
Fig. 5: Results
Need to collect more answers for a CrowdFlower survey?
If your survey is complete and you would like to collect more information, here are the steps to add more judgments to your survey:
- Go to the Data page and select 'Download'. Remove the column '_unit_id'' and then re-upload that data set to your job
- Click 'Proceed Anyway' if you see the following message:
- Set the number of judgments you need on the Launch page
- Select 'Order more rows'
You will not collect judgments from contributors who have already accessed your survey this way.
Thanks for reading our tutorial. We hope you found it useful and insightful. If you have any further questions or concerns, please refer to our CML documentation on the Success Center. Once again, we stress that we are not responsible for any discrepancies in payment as we do not officially support using the platform in this manner.
If you believe your survey could take more than 30 minutes to complete, please email firstname.lastname@example.org so we can assist you in setting up your survey. The CrowdFlower platform currently does not support surveys that require more than 30 minutes.