Judgment Overcollection FAQ

In some cases, rows may collect more judgments than you expected. This article answers the question "Why might my rows have more judgments than I thought?"


If a job has an auto-launch threshold and there aren't enough rows to fill a page, rows that are already finalized will be used to fill a page.

Example: a job has an auto launch timeout of 5 minutes. There are 10 rows per page. There is only one unordered row. CrowdFlower will create a page with that one unordered row and 9 already finalized rows. Those already finalized rows will collect more judgments than specified because that one unordered row needs judgments.

Forgiving Test Questions

If the correct answer for a Test Question is changed, contributors have their scores recalculated. If those contributors were untrusted before and were actually correct on those Test Questions, they can become trusted. When they become trusted, the rows they judged now have an additional trusted judgment which may lead to them having "too many" trusted judgments.

No Quiz Mode

When there's no quiz mode, contributors can judge lots of rows before seeing enough Test Questions to become trusted. Because CrowdFlower only counts trusted judgments on a row, if after a certain time contributors become trusted, those judgments that were previously untrusted become trusted, and the row can overcollect judgments.

Looking at the Data tab

On the data tab (jobs/{{job_id}}/units), the judgments displayed are for both trusted and untrusted judgments. A user sets how many trusted judgments they want in a job in the Job Settings menu.

Looking at an Individual Unit

The judgments here are total judgments. Click on "Show Worker Info" to see how many workers are trusted.


Variable Judgments Mode

Variable Judgments allows the user to request a range of judgments in the hopes of achieving a minimum confidence on each row. For example, if a job is set to collect 3 judgments per row and dynamic judgments mode is turned on with max judgments per row set to 6, then rows should collect 3-6 trusted judgments.

Units Remain Finalized - Job undercollecting judgments

If a job has "rows remain finalized" checked, rows can potentially undercollect.

Example: if a job is set to collect 3 judgments / rows and has already collected that many judgments, if one of those contributors later becomes untrusted because they fall below the trust threshold or are rejected by our anti-scamming software, then that judgments will only show 2 trusted judgments.

Was this article helpful?
0 out of 0 found this helpful

Have more questions? Submit a request
Powered by Zendesk