Follow

Guide to: Reports Page and Options Page

Reports may be generated throughout the lifespan of your job to provide you with accurate information on the data you have uploaded, the performance of your crowd, and the results they submitted. The Reports page can be accessed by clicking on the Results in the top navigation. For this article, please note that the term "unit" refers to row and "gold" refers to Test Questions.

Results

Screen_Shot_2017-05-09_at_12.15.10_PM.png

Fig. 1: Reports Page

 

Accessing Reports

After your job has begun, the Results Tab will display several options, each with the up-to-date, real time collections of results the job has collected so far. You are able to download any of these reports by simply clicking the Download button. 

Options

Click on the 'Options' tab to the right of the 'Results' tab to customize the data included in the reports.

Screen_Shot_2017-05-09_at_12.18.05_PM.png

Fig. 2: Options Page

Aggregated Results Settings

Fig. 3: Aggregation Options for Each Result Column

The aggregation settings allow you to select the aggregation method for each question in your job. Each question will result in a column generated by the job with the set aggregation:

  • Best Answer ('agg') - Returns the highest confidence response
  • All Answers ('all') - Returns every single response inputted
  • Numeric Average ('avg') - Returns a numeric average calculated based on all responses
  • Top # Answers ('agg_x') - Returns the top 'x' responses
  • Bounding Box Aggregation ('bagg_x') - For bounding box image annotation jobs only, returns box responses that overlap above 'x' Intersection over Union (number between 0 and 1)
  • Confidence Aggregation ('cagg_x') - Returns the answers that are above 'x' confidence (number between 0 and 1)

See this article for more information about aggregation.


 

Report Types

There are six types of reports that can be generated and downloaded after the job has begun running:

Full Report 

This report lists each individual response from contributors as a separate row, along with other pertinent information such as IP address, Contributor ID, Trust Score and Channel. This report will generate automatically upon completion of the job.  It contains all of your original source columns, as well as:
  • _unit_id: A unique ID number created by the system for each row
  • _created_at: The time the contributor submitted the judgement 
  • _golden: This will be "true" if this is a test question, otherwise it is "false"
  • _id: A unique ID number generated for this specific judgment
  • _missed: This will be "true" if the row is an incorrect judgment on a test question.
  • _started_at: The time at which the contributor started working on the judgement
  • _tainted: This will be "true" if the contributor has been flagged for falling below the required accuracy. This judgment will not be used in the aggregation.
  • _channel: The work channel that the contributor accessed the job through
  • _trust: The contributor's accuracy. Learn more about trust here
  • _worker_id: A unique ID number assigned to the contributor
  • _country: The country the contributor is from
  • _region: A region code for the area the contributor is from
  • _city: The city the contributor is from
  • _ip: The IP address for the contributor
  • {{field}}: There will be a column for each field in the job, with a header equal to the field's name.
  • {{field}}_gold: The correct answer for the test question

Note: The term "unit" refers to row


Aggregated Report

This report includes aggregates all of the responses for each individual row in the job. Each row will aggregate according to the method defined under the Reports tab on the Results page. If you select 'Include Confidence Summary,' each response will be paired with a confidence score, or the agreement weighted by trust for all answers given. It contains all of your original source columns, as well as:

  • _unit_id: A unique ID number created by the system for each row

  • _golden: This will be "true" if this is a test question, otherwise it is "false"

  • _unit_state: This will be "finalized" if the row has collected all judgments needed, "new" if it has not been launched yet, "judgable" if it requires more judgments, and "golden" if the row is a test question.

  • _trusted_judgments: The number of non-tainted judgment the row has accumulated

  • _last_judgment_at: The time the latest judgment was received

  • {{field}}: There will be a column for each field in the job, with a header equal to the fields name

  • {{field}}:confidence: If you choose to include confidence values, these columns will be included. They represent the level of agreement between contributors.  For more on calculating confidence scores, see here.


Source Report

This report includes the original, unprocessed data that was uploaded to the job before it was run. If you do not select 'Exclude Test Questions,' this report will also include Test Question answers and reasons created for a job.

  • _unit_id: A unique ID number created by the system for each row

  • _created_at: The time the row was first uploaded into the job

  • _updated_at: The time the row was most recently changed


Test Questions Report 

This report includes only data on the test question rows in the job.

  • _id:  A unique ID number created by the system for each test question

  • _pct_missed: Percentage of responses that were incorrect

  • _judgments: Total number of judgments this test question received

  • _hidden: This will be "true" if the test question is disabled

  • _contention: The contentions from contributors, separated by newline

  • _pct_contested: Percentage of contributors who answered test question incorrectly and contested

  • _gold_pool: This will be "quiz" for test questions set only for quiz mode, "work" for test questions set only to work mode, and blank for test questions set to both

  • {{field}}: There will be a column for each field in the job, with a header equal to the field's name

  • {{field}}_gold: The correct answer for the test question

  • {{field}}_gold_reason: The reason/explanation for the correct answer

Note: The term "gold" refers to Test Questions


Contributor Report

This report contains information on the performance of individual contributors in the job.

  • worker_id: A unique ID number assigned to the contributor

  • external_id: A unique ID number assigned to the contributor from external channels

  • judgments_count: Number of judgments submitted by contributor

  • missed_count: Number of test questions answered incorrectly by contributor

  • golds_count: Number of test questions answered by contributor

  • forgiven_count: Number of test questions answered incorrectly by contributor, that were then forgiven

  • feedback_count: Number of times contributor left a contestation

  • frontload_count: Number of test questions seen by contributor in quiz mode

  • frontload_missed: Number of test questions answered incorrectly by contributor in quiz mode

  • channel: The work channel that the contributor accessed the job through

  • country: The country the contributor is from

  • region: A region code for the area the contributor is from

  • city: The city the contributor is from

  • llast_ip: The most recent IP address for the contributor

  • flagged_at: The time at which the contributor was flagged

  • rejected at: The time at which the contributor's answers were rejected

  • bonus: The amount the contributor has been bonused

  • flag_reason: The requester inputted reason the contributor was flagged

  • trust_overall: The trust level of the contributor

  • submission_rate: The average speed at which the contributor submitted judgments

  • {{skill_tag}}: There will be a column for each skill tag required, with a "1" for each contributor that possesses that skill tag

Note: The term "gold" refers to Test Questions


JSON Report

This report contains JSON formatted data for each judgment and row. You will most likely need to use a JSON parser in conjunction with this report. 

 

 


Was this article helpful?
2 out of 2 found this helpful


Have more questions? Submit a request
Powered by Zendesk