Follow

Guide to: Reports Page and Settings Page

Reports may be generated throughout the lifespan of your job to provide you with accurate information on the data you have uploaded, the performance of your crowd, and the results they submitted. The Reports page can be accessed by clicking on the Results in the top navigation. For this article, please note that the term "unit" refers to row and "gold" refers to Test Questions.

Results

Fig. 1: Reports Page

Only the Source Data File Will Appear For Download Before Job Begins

Before the job launches, the Reports Page will display the Source Data as an option available for download. The Source Data refers to data you uploaded to the job. If you have not uploaded data to the job yet, the Source Data file will be empty. If you have created or uploaded Test Questions, the Test Question report will also be available.

More Job Reports Available After Your Job Has Launched

After your job has begun, the Reports Tab will display several more options, each with the up-to-date, real time collections of results the job has collected so far. You are able to download any of these reports by simply clicking the Download button. 

When your job completes, the platform will automatically generate the Full and Aggregated Reports.

Settings

Click on the 'Settings' tab to the right of the 'Results' tab to customize the data included in the reports.

Fig. 2: Settings Page

Aggregated Results Settings

Fig. 3: Aggregation Options for Each Result Column

The Aggregation Settings allow you to select the aggregation method for each question in your job. Each question will result in a column generated by the job, and you can select the aggregation method for each of these columns.

  • Best Answer ('agg') - Returns the highest confidence response
  • All Answers ('all') - Returns every single response inputted
  • Numeric Average ('avg') - Returns a numeric average calculated based on all responses
  • Top # Answers ('agg_x') - Returns the top 'x' responses
  • Confidence Aggregation ('cagg_x') - Returns the answers that are above 'x' confidence (number between 0 and 1)

See this article for more information about aggregation.

 

There are several checkboxes that allow you to set which information to include in your report.

  • Include Untrusted Judgments in Full Data
    • Include untrusted judgments in the Full Report
  • Exclude Test Questions
    • Exclude test questions as rows in the reports
  • Include Unfinished Rows
    • Include all of the rows that have received one or more judgments. If you do not check this box, only the rows that have finalized by collecting the number of judgments that you specified in the Job Settings will be included in the reports
  • Logic Aware Aggregation
    • Aggregation is based on only-if logic in your job design. If a contributor does not provide judgments that follow the only-if logic, his/her judgments are not included in the confidence scores.

 

 


 

Report Types

There are six types of reports that can be generated and downloaded after the job has begun running:

Full Report 

This report lists each individual response from contributors as a separate row, along with other pertinent information such as IP address, Contributor ID, Trust Score and Channel. This report will generate automatically upon completion of the job.  It contains all of your original source columns, as well as:
  • _unit_id: A unique ID number created by the system for each row
  • _created_at: The time the contributor submitted the judgement 
  • _golden: This will be "true" if this is a test question, otherwise it is "false"
  • _id: A unique ID number generated for this specific judgment
  • _missed: This will be "true" if the row is an incorrect judgment on a test question.
  • _started_at: The time at which the contributor started working on the judgement
  • _tainted: This will be "true" if the contributor has been flagged for falling below the required accuracy. This judgment will not be used in the aggregation.
  • _channel: The work channel that the contributor accessed the job through
  • _trust: The contributor's accuracy. Learn more about trust here
  • _worker_id: A unique ID number assigned to the contributor
  • _country: The country the contributor is from
  • _region: A region code for the area the contributor is from
  • _city: The city the contributor is from
  • _ip: The IP address for the contributor
  • {{field}}: There will be a column for each field in the job, with a header equal to the field's name.

Note: The term "unit" refers to row


Aggregated Report

This report includes aggregates all of the responses for each individual row in the job. Each row will aggregate according to the method defined under the Reports tab on the Results page. If you select 'Include Confidence Summary,' each response will be paired with a confidence score, or the agreement weighted by trust for all answers given. It contains all of your original source columns, as well as:

  • _unit_id: A unique ID number created by the system for each row

  • _golden: This will be "true" if this is a test question, otherwise it is "false"

  • _unit_state: This will be "finalized" if the row has collected all judgments needed, "new" if it has not been launched yet, "judgable" if it requires more judgments, and "golden" if the row is a test question.

  • _trusted_judgments: The number of non-tainted judgment the row has accumulated

  • _last_judgment_at: The time the latest judgment was received

  • {{field}}: There will be a column for each field in the job, with a header equal to the fields name

  • {{field}}:confidence: If you choose to include confidence values, these columns will be included. They represent the level of agreement between contributors.  For more on calculating confidence scores, see here.


Source Report

This report includes the original, unprocessed data that was uploaded to the job before it was run. If you do not select 'Exclude Test Questions,' this report will also include Test Question answers and reasons created for a job.

  • _unit_id: A unique ID number created by the system for each row

  • _created_at: The time the row was first uploaded into the job

  • _updated_at: The time the row was most recently changed


Test Questions Report 

This report includes only data on the test question rows in the job.

  • _id:  A unique ID number created by the system for each test question

  • _pct_missed: Percentage of responses that were incorrect

  • _judgments: Total number of judgments this test question received

  • _hidden: This will be "true" if the test question is disabled

  • _contention: The contentions from contributors, separated by newline

  • _pct_contested: Percentage of contributors who answered test question incorrectly and contested

  • _gold_pool: This will be "quiz" for test questions set only for quiz mode, "work" for test questions set only to work mode, and blank for test questions set to both

  • {{field}}: There will be a column for each field in the job, with a header equal to the field's name

  • {{field}}_gold: The correct answer for the test question

  • {{field}}_gold_reason: The reason/explanation for the correct answer

Note: The term "gold" refers to Test Questions


Contributor Report

This report contains information on the performance of individual contributors in the job.

  • worker_id: A unique ID number assigned to the contributor

  • external_id: A unique ID number assigned to the contributor from external channels

  • judgments_count: Number of judgments submitted by contributor

  • missed_count: Number of test questions answered incorrectly by contributor

  • golds_count: Number of test questions answered by contributor

  • forgiven_count: Number of test questions answered incorrectly by contributor, that were then forgiven

  • feedback_count: Number of times contributor left a contestation

  • frontload_count: Number of test questions seen by contributor in quiz mode

  • frontload_missed: Number of test questions answered incorrectly by contributor in quiz mode

  • channel: The work channel that the contributor accessed the job through

  • country: The country the contributor is from

  • region: A region code for the area the contributor is from

  • city: The city the contributor is from

  • llast_ip: The most recent IP address for the contributor

  • flagged_at: The time at which the contributor was flagged

  • rejected at: The time at which the contributor's answers were rejected

  • bonus: The amount the contributor has been bonused

  • flag_reason: The requester inputted reason the contributor was flagged

  • trust_overall: The trust level of the contributor

  • submission_rate: The average speed at which the contributor submitted judgments

  • {{skill_tag}}: There will be a column for each skill tag required, with a "1" for each contributor that possesses that skill tag

Note: The term "gold" refers to Test Questions


JSON Report

This report contains JSON formatted data for each judgment and row. You will most likely need to use a JSON parser in conjunction with this report. 

 

 


Was this article helpful?
1 out of 1 found this helpful


Have more questions? Submit a request
Powered by Zendesk