Follow

Guide to: Converting Finalized Rows into Test Questions

Test Questions are the most important quality control mechanism in the CrowdFlower Platform. They not only ensure the most capable contributors are working on your job, creating more of them can lead to increased throughput. This feature allows you to convert already completed rows into test questions quickly and effectively. It also helps to create test questions with an appropriate answer distribution, as you can filter for the questions and answer options you’d like to convert. Using this feature allows you to create more, high quality test questions in significantly less time! As well, test questions may also be converted in a report by follow the instructions in this article.

Note: This process is best used on a job that has successfully launched 100 rows with quality results. These 100 rows can then be converted into test questions. They can also be downloaded in a test question report and uploaded to a new job with no test questions via the Data page.

Steps to convert rows

  1. Open job with at least 100 finished rows
    • When a job run has completed, the Monitor page will show a new message (shown below) alerting you that completed rows are available to be converted into test questions.

Fig.1: Job with at least 100 finished rows showing message to convert rows

2. Initiate conversion of test questions

    • Clicking the link on the Monitor page will start the converting of the possible rows. An email will be sent to the user who selects this link when the indexing has completed. The job owner will not be notified of the conversion.

 

Fig. 2: The conversion has started and an email will be received when completed

3. Convert rows via the UI

  • When the indexing has finished, the test question conversion interface can be accessed via the link in the email or through the Monitor page (after refreshing).

 

Fig. 3: Converting rows have been completed and link to access interface

 

Fig. 4: Test question conversion interface

  • A. Selected question filter
    • Here you are able to select the specific question you’d like to view
      • Items to note:
        • You may only view one question and it’s answer at a time
        • When filtering on a logic dependent question, you must first select the appropriate answer to the parent question before converting
  • B. Answer to selected question filter
    • After selecting a question to filter, you may then select an answer to filter by
      • Items to note:
        • Textbox answers are not able to be filtered on, but can be answered along with the other fields on the page using the ‘+’ button next to the answer you’d like to select
        • When filtering, only the top answer will be shown as a possible answer to the question
        • Filtering by the answer to the question will help you create an appropriate answer distributions that ensures your testing for all possible cases and not biasing contributor to a particular answer
  • C. Next/Skip button
    • Here you can skip a question if you prefer to not convert it to a test question
  • D. Convert row to test question button
    • After finding a row that is a good fit for a test question, and selecting all appropriate answers, click this button to convert into a test question
  • E. Number of total rows available
    • This number shows you the total amount of completed rows available to be converted
  • F. Answer distribution
    • The answer distribution for already created test questions can be viewed here

Converted rows

After converting the selected rows you would like, they will appear in the list of test questions from the Quality page.

Be sure to check the Answer Distributions modal (accessible from the box on the right side of the page) to help you determine which answers are under-represented in your set of Test Questions. 
 
To avoid biasing contributors, the ideal answer distribution for a job is somewhere between your source data set's true answer distribution, and an even distribution across each possible answer.
 
For example, there are 3 possible options (say, Positive, Neutral and Negative), and you know that your data seems to end up about 70% Positive, 10% Neutral and 20% Negative. In this case, a good Test Question answer distribution would be between 70%-50% for Positive 10%-25% each for Neutral and Negative.

If you would like to instead covert high confidence rows in a .csv please check out this article.


Was this article helpful?
0 out of 0 found this helpful


Have more questions? Submit a request
Powered by Zendesk