Test Questions are the most important quality control mechanism in the CrowdFlower Platform. Creating ideal Test Questions is the best way to ensure high quality results from a job. This article explains the characteristics of useful Test Questions and best practices when creating them.
How many Test Questions do I need?
Most jobs have between 50-100 test questions however the more test questions in a job the faster the job will complete. By default there is one Test Question per page (with 1-19 rows per page set) which means a contributor can only do as many pages of work as there are Test Questions in a job. Quiz Mode will also have effect on the amount of Test Questions you need to create. If you create 15 total Test Questions and have 10 Rows Per Page, Quiz Mode will first use 10 of these Test Questions. From there you will have 5 Test Questions remaining and contributors can only do 5 more pages of work before maxing out. 50 total judgements are collected per contributor but as there are 1 Test Question per page, 5 (1 per page) will be Test Questions. Therefore, the more Test Questions you have in your job, the more work any individual contributor can complete.
Characteristics of Ideal Test Questions
- A good Test Question must be correct and complete with respect to the job instructions. The answers given must be more objective than subjective. Be careful to not answer questions based on prior knowledge about the data in the job. Each Test Question reason should be able to explain to any contributor exactly how that answer was reached.
- A good Test Question allows all possible answers. It’s a better experience for contributors if your Test Questions are a little more lenient than overly strict.
- Example: The job is to rate the sentiment of a tweet. In this type of job, what someone understands as Slightly Positive, others may think is Highly Positive. Accepting both answers will allow contributors to get the Test Question correct when either answer is selected.
Fig. 1: Both Highly Positive and Slightly Positive are enabled as correct answers
- Test Questions should have an adequate difficulty level. Test Questions should be hard enough to test a contributor’s performance but easy enough for those workers who are trying to follow your instructions honestly. Not all data rows make for great Test Questions. Don't create Test Questions that take you significantly more time than an average data row to complete or that don't have a correct response in regards to the instructions.
- Every Test Question should be equipped with an instructive reason that explains the process to getting a correct answer.
- Write full sentences and use proper grammar
- Refer directly to the instructions
- Explain why the answer is correct and also why the incorrect answers are not. Find points for them to improve in their work
- Write long, complicated sentences. Try to use simple vocabulary.
- Be careful when writing witty/snarky reasons. A little humor can lighten the mood, but good contributors might feel insulted if your reasons seem too sarcastic.
- Never leave a reason blank. Only having a single word or overly generic reasons are also not helpful to the contributors.
- Ideally, contributors will behave exactly the same on Test Questions as they do on the regular rows of data in your job. To accomplish this, your Test Questions must be indistinguishable from the overall data set. Be sure that the Test Questions in a job reflect the source data and remain ‘hidden’ throughout the job to keep the contributor honest.
- Test Questions should have an appropriate Answer Distributions that reflect your dataset. An even Answer Distribution will train contributors on every possible answer instead of biasing them towards one answer.
- The method of answering Test Questions varies if your job contains many questions to be answered or just a few.
- For simple jobs that ask only a few questions, it’s a good idea to define an answer for every field.
- For complicated jobs that ask a lot of questions and include logic, you should vary which questions you answer within your test questions. Varying your answer makes it easier for contributors to get the question correct because they are tested on less fields. If contributors get one question wrong they get the entire Test Question wrong. Also, switching which questions you answer will keep contributors honest as they won’t know which questions they’ll be tested against for a particular Test Question.
- If you are attempting to submit impossible logic combinations in your test questions, you will see a message appear. This will warn you that a piece of the logic tree is not correct and you will need to update the question before saving.
Fig. 2: Warning message in regards to answering the first question as ‘No’ and then answering the second question which should only be asked if the answer to the first question is ‘Yes’
Question 1: Does this tweet mention American Airlines?
- Answer 1a: Yes
- Answer 1b: No
Only asked if the answer to Question 1 is ‘Yes’:
Question 1: Does this tweet mention American Airlines?
- Answer 2a: Positive
- Answer 2b: Neutral
- Answer 2c: Negative
Answer ‘Yes’ to Question 1 and answer any choice from Question 2
Answer ‘No’ to Question 1 and answer any choice from Question 2. In this scenario, the contributor would not be shown Question 2 and therefore get the Test Question wrong without ever having the chance to answer it.