Overview
Quality Flow projects can collect between 1 - 20 judgments per row.
When collecting more than one judgment per row, there are some important notes to be aware of:
- Once a job has been set to running at lease once, the number of judgments to collect cannot be changed.
- It is possible to collect more that one judgment for unit groups, but in this case you only have the option to retain groupings.
- Only work jobs can collect multiple judgments
- Work jobs collecting multiple judgments are not currently available for following QA job creation. This feature is coming soon.
View Multiple Judgment Results
- On the DATASET summary there are two views: unit view and judgment view. Unit view allows you to browse and manage the source data easily. If a unit is collecting multiple judgments, all judgement-related information will be collapsed into one row of the same unit.
- After switching to Judgment view, each judgment will be visible in its own row with a unique judgment number.
- Column "Max judgments" means the max number of judgments the job is set to collect.
- Column "Judgments collected" means the number of judgments the unit has collected so far.
- In the Unit column the "Status" refers to the unit status. In the Latest Judgment column, the status is at judgment level. This means you can track each contributor's progress on different judgments.
- When a unit has not collected any judgments yet the unit status is JUDGABLE. When a unit has collected the max number of judgments, the unit status is SUBMITTED. When a unit has collected at least one judgment, but has not collected the max number of judgments yet, its status is WORKING.
- The DATASET summary displays the total number of units and total number of judgments to be collected in the top row. Click on TOTAL UNITS or TOTAL JUDGMENTS to view the relevant progress numbers.
Download Multiple Judgment Reports
- In the Dataset download, by deafult, each judgment is given in a separate row with a unique judgment number.
- It is also possible to download aggregated results at the Job level under Job > Results.
- Four aggregation methods are supported:
- Best answer ('agg') - Returns the top 1 response.
- All answers ('all') - Returns all responses.
- Numeric average ('avg') - Returns a numeric average calculated from all results.
- Top # Answers ('agg_x') - Returns the top 'x' responses.