Follow

Quality

Lesson Overview:

  • This lesson goes over Test Questions, which are CrowdFlower’s quality control mechanism.
    • Test Questions are created by the job author to:
      1. Qualify high performing contributors and remove low performing contributors
      2. Continually train contributors and improve their understanding of the job
    • Test Questions are used to rate contributor’s accuracy on your job
      1. When contributors answer a test question, their answer is checked against that of the job owner. If their answer doesn’t match, their job accuracy goes down. If their accuracy at any point falls below the minimum accuracy threshold, they will be removed from the job and their judgments will not be included in the final output. Instead we’ll replace their judgments with the work of a contributor that has maintained an accuracy above the minimum threshold.
    • Quiz Mode Test Questions:
      1. In order for contributors to enter your job, they must pass a quiz compiled from the test questions you created.
      2. All questions in Quiz Mode are test questions.
      3. Quiz mode is only activated if you have created enough test questions.
    • Work Mode Test Questions:
      1. In Work Mode contributors will begin enriching rows from your source data by completing them in ‘pages’ which consist of 5-10 source rows at a time.
      2. Every page in Work Mode will contain 1 Test Questions, however contributors will not know which row its is.
    • More Test Question = Higher Throughput
      1. We limit the amount of work a contributor can complete by the number of test questions that are in the job.
      2. Once a contributors has seen all of the Test Questions in your job, they ‘max out’ and cannot submit any more work.
      3. The more test questions there are in a job, the more work each contributor can do which increases overall throughput A recommended number of test questions for your job will be suggested on the quality page

Test Question Report Column headers:

  • _golden: This value must be set to TRUE for all test question rows.
  • relevant_yn_gold: This is either yes or no for each test question.
  • relevant_yn_gold_reason: This is our reasoning for the above answer.
  • sentiment_gold: Our sentiment rating for the tweet.
  • sentiment_gold_reason: Our reasoning for why we chose the sentiment rating.
  • likes_gold: Our answer for the number of likes.
  • likes_gold_reason: Our reasoning for this answer.

Steps To Complete:

  1. Navigate to the ‘Quality’ page
  2. Select ‘Create Test Questions’
  3. Create 3 test questions
  4. Download the Test Question Report here
  5. Navigate to the data page of your job
  6. Click “Add More Data”
  7. Upload the Test Question report
    • Note that if you add data that includes extra columns, or is missing a column that already exists in the job, you will get a message similar to the one below. This is just to notify you that your column headers do not match those in the job, in case you’ve selected the wrong file to upload. If the file is indeed correct, you can click “Proceed Anyway” to continue the upload.
  8. Once all new rows are uploaded, click “Convert Uploaded Test Questions”
  9. Navigate to the ‘Quality’ page and ensure you have 20 new test questions

Further Reading:

Next Lesson: Settings & Launch


Was this article helpful?
0 out of 0 found this helpful


Have more questions? Submit a request
Powered by Zendesk