How to: Create Test Questions

Test Questions are the most important quality control mechanism in the CrowdFlower Platform. Test Questions are rows that are pre-labeled with known answers and are randomly inserted throughout your job. These will be used to test and track contributor performance. Test Questions ensure that only the contributors demonstrating competency in your job are allowed to submit judgments. Learn characteristics of ideal Test Questions in the Test Question Best Practices article.

Why use Test Questions?


Test Questions will be used to test a Contributor’s performance both before they enter a job (in Quiz Mode) and on an on-going basis while they work in a job. Test Questions allow you to understand the performance of each contributor that works on a job and automatically remove contributors that perform poorly. By setting a accuracy threshold in a job, it will only allow contributors who have performed above the threshold to complete any further work on a job.

Note: Set accuracy threshold Settings > Quality Control > Test Question Settings. These settings as well as others are described in detail in the Test Question Settings article.


An important step in creating Test Questions is to write very clear reasons explaining why you have selected that answer. These reasons will be shown to contributors if they get a Test Question incorrect. This is a great tool to explain to your contributors how you’ve reached your answer. By explaining the answer, contributors can learn the rules and intricacies of a job in more depth than is provided in the instructions.

Creating Test Questions

Here we will go through the steps to creating Test Questions with our interface, you can read more about the specifics of this page here.

Fig. 1: Test Question creation interface

Steps to Creating Test Questions:

    • Make sure you have your CML design completed and your source data uploaded
      • Note: It is important that you don't change the structure of your CML after you create test questions, so make sure you have finalized your job design.
    • Go to the Quality tab or "Next: Create Test Questions" button in the top right corner of the Design page

Fig. 2: To get to the Test Question creation interface, select either of these options

    • From here, click “Create Test Questions” in the center of the page.

Fig. 3: Quality page with no existing Test Questions

    • Proceed through each of the rows, selecting those of which you wish to use as Test Questions
    • Finally, answer the rows you selected as Test Questions correctly and provide a reason for each answer.

Fig. 4: Test Question answered with reasons

    • Note: When you provide a Test Question reason, make sure it mentions how the correct answer was found and tips on how to identify the correct answer. Explaining how an answer was determined can go a long way in teaching your crowd how to provide you with the best responses.

Choose your Test Questions wisely

To achieve an even answer distribution of Test Question answers, you can choose not to answer a presented row and skip it instead. This is also useful for rows that would not make helpful Test Questions due to subjectivity or difficulty. You can choose to skip a row if you do not want to turn it into a Test Question by clicking the Skip button.

Make sure to check your answer distributions throughout the Test Question creation process as to not bias certain answers. You should try your best to have an even distribution for every question. For example, if you have a question “Is this posting relevant to American Airlines?” you will want to have the distribution as close to 50% “Yes” and 50% “No” as possible. If you are having difficulties creating an even distribution and have similar job to the one you are currently designing, you can create Test Questions from high confidence rows.

Fig. 5: The Skip button and Answer Distributions

Manual Test Questions

It is possible to create Test Questions in bulk via a data file by uploading all of your Test Questions at the same time. You can also update existing Test Questions in bulk using a Test Question Report.

Upload New Test Questions (Via Data Page)

It is possible to upload Source Data that already contains Test Questions.


To accomplish this the Source Data must include columns:

  • {fieldname_gold}: Test Question answer - lower-case with underscores instead of spaces
  • {fieldname_gold_reason} Test Question reason - plain text
  • {_golden} - set to ‘TRUE’ for each Test Question

Fig. 6: Source Data with ‘ _golden’, ‘cf_sentiment_gold’, ‘cf_sentiment_gold_reason’


  1. Go to the Data page within your job
  2. Select “Add More Data”
  3. A message may appear letting you know that your column headers do not match the data already uploaded to the job. Select ‘Proceed Anyways’ as we know this is a Test Question report and does not follow the data exactly
  4. Once your test questions have been added to your data page, click “Convert Uploaded Test Questions”
  5. Go to the Quality page to see your newly uploaded Test Questions

Tip: Test Questions can also contain multiple correct answers. This can occur in scenarios where the response of the contributor is more subjective than objective. When specifying Test Questions manually, multiple values in the Source Data will be determined by the newline character.

Fig. 7: Column ‘cf_sentiment_gold’ all have multiple values stored in one cell separated by the newline character

Update Existing Test Questions (via Quality Page)

If you would like to batch update Test Questions already in your job, you may do so via a Test Question report.


    • 1. Go to the Quality page and select ‘Download test question report’

Fig. 8: Test Question report location


  • 2. Edit Test Question answers (any fieldname_gold columns)
    • Note: Do not edit the ‘_id’ column as this will reference each specific Test Question upon upload
  • 3. Select ‘Upload training data as test questions’ from the Reports drop down menu
  • 4. Upload edited report
    • Note: The page will automatically refresh with updated Test Questions

Was this article helpful?
0 out of 0 found this helpful

Have more questions? Submit a request
Powered by Zendesk