Follow

Spotcheck (Beta) Feature

The in-platform Spotcheck (beta) feature automates the selection of a random set of rows and the process of report generation for auditing job results. This feature is available in an invite-only beta for selected customers with a license. Contact your Customer Success Manager for more details.

Initiating Spotcheck

The Spotcheck feature can be initiated on the Results page using the "Use Spotcheck" action.

Screen_Shot_2017-12-07_at_10.31.15_AM.png

This operation creates a copy of the job titled "Spotcheck:{Job_Name}" with a sample of hundred randomly selected rows and their judgements.

Screen_Shot_2017-12-07_at_10.30.32_AM.png

You can begin reviewing results on the Quality page in a process that is similar to creating test questions. When you begin to convert a row to a test question, you will see contributor responses for the row. At this stage, you don't have to worry about those responses, just record and save your response for each row.

Screen_Shot_2017-11-10_at_8.23.06_AM.png

We recommend that you create atleast 20-30 test questions to gain insight into your data quality. However, the more rows you audit, the more accurate your estimate will be.

NOTE: Spotcheck jobs already have the data required to complete spotcheck. You don't need to upload data or modify the job design in order to complete spotcheck successfully.

Downloading Spotcheck Results

Screen_Shot_2017-12-07_at_10.31.54_AM.png

Once you have created the desired number of rows you want to audit, you can download the audit summary on the Reports page in the form for the Spotcheck Report.

The Spotcheck report contains your responses recorded as part of the test question creation process, the aggregated result from all contributor responses as well as the agreement between the 2.

Screen_Shot_2017-11-10_at_8.40.23_AM.png

Columns in the report

  • question_name:gold_answer - Your response during test question creation in the spotcheck job.
  • question_name:answer - Aggregate of contributor responses from the original job.
  • quesiton_name:confidence - Confidence for the aggregate response.
  • quesiton_name:correct? - Determines if the aggregate response matched your response, gold_answer.

You can compute the percentage or correct / incorrect answers from the "question_name:correct?" column.

NOTE: Spotcheck currently does not support Image Annotation usecases. You can audit results from these jobs using the manual audit process.


Was this article helpful?
0 out of 0 found this helpful


Have more questions? Submit a request
Powered by Zendesk