This tutorial will show you how to create and run a Sentiment Analysis job on the CrowdFlower Platform.
Note: The product, Senti, is no longer supported on the CrowdFlower platform. Please follow this tutorial to create a Sentiment Analysis job.
We have our trained crowd grade every piece of content on a 5 point sentiment scale in relation to your relevance question:
- 5 = Very Positive
- 4 = Slightly Positive
- 3 = Neutral
- 2 = Slightly Negative
- 1 = Very Negative
Our crowd picks up on sarcasm, nuance, and idiomatic English (i.e. "that song is dope!") that automated solutions can't.
In addition to finding top-level relevance, you can ask more detailed "topic-level" questions. The questions are in the form of "Does this post talk about __" where you fill in the blank. Please also provide context around your question that a non-expert on the topic could use to answer it. Topic- level questions are limited to 300 characters.
If you wanted to know if people had seen a movie, you would write in "that the person has already seen the movie", and we display to the crowd "Does the posting mention that the person has already seen the movie?" a few other examples are:
It is important that your questions are specific as possible and accessible to general public so that our trained crowd can answer them.
Steps to running a successful Sentiment Analysis job
There are eight basic steps to running jobs:
- Choose a template: Choose the "Sentiment Analysis" template.
- Customize your Job: Use the job editor to customize the interface that contributors will use to complete the job.
- Upload your Data: Upload the data that you want contributors to process.
- Preview your Job: View the job from the contributor's perspective.
- Create Test Questions: Use the UI to answer a set of your own questions.
- Launch your Job: Send the job to the crowd.
- Obtain Judgments: Monitor and correct Test Questions as appropriate.
- Retrieve and Interpret your Results: Complete your job, at which point results are automatically compiled, stored and aggregated.
1. Choose a template
Once you have logged in to your platform account, click "Create Job". You will be prompted to choose a template from the library.
Choose "Sentiment Analysis" or click "View All Templates" to choose the advanced version "Judge the Relevance and Sentiment of content about (a topic)" for relevance and topic sentiment coding.
Fig. 1: Choose a template
On the top right, choose "Use This Template."
2. Customize your Job
Once the template has been chosen, it's time to customize the job to your dataset.
In the "Overview" section of the instructions and in the questions of the job, replace "(the topic)" with the topic in your dataset.
Fig. 2: Customize your Job
Add any additional instructions or context that might help contributors in the instructions.
3. Upload your Data
First, click the Data tab on the left side of the platform to be prompted to upload data.
Fig. 3: Upload your Data
Next, choose "Upload File" and select the dataset for your job.
Note: CSVs need to be UTF-8 encoded
4. Preview your Job
To ensure the job is as you expect, click the "Preview" button on the left side of the page. This will open a preview of your job which shows exactly what contributors will work on.
Fig. 4: Preview your Job
5. Create Test Questions
Click Test Questions on the left side of the platform to be prompted to create Test Questions. This will allow you to answer some of your own rows that can be used to test contributor performance on a job.
Visit the How-to Create Test Questions article to learn how to create Test Questions.
The Test Question Best Practices article explains the characteristic of great test questions.
6. Launch your Job
Before you launch the job, it is advantageous to use the Job Launch Checklist to ensure everything has been considered in the settings of your job.
When you have completed the checklist, click Launch on the left navigation panel. From this page, you can also specify the number of rows from your entire dataset you would like to order.
You will be prompted with the cost of the job you wish to run and an option to add credits if you lack sufficient funds in your account to cover the amount displayed.
Fig. 5: Launch your Job
7. Obtain Judgments
From the moment you launch your job, you can use the dashboard on the Monitor page to check on the status of your job and quantify the number of judgments rendered.
8. Retrieve and Interpret your Results
While you might wait until you job is completed to review your job’s results, you can generate and download your results at any time by navigating to the Reports page.
There are six reports to choose from:
This report includes every judgment made by every contributor. It contains (among other things) contributor IDs, trust rating, labor channel, and contributor judgments. You'll use this CSV to dive deeply into the data to understand patterns (e.g., maybe all contributors from a specific region are providing poor responses, but you wouldn't be able to tell that from the Aggregated CSV).
The aggregated report includes one result for every row. Since multiple contributors answer each row, the aggregated CSV aggregates contributor's judgments based on individual contributors' trust ratings. This CSV is what you are likely to interact with the most.
This is the original source data from the data file you uploaded earlier.
The Test Question report includes data on the responses gathered by your Test Questions.
This report includes data on the contributors who participated in the job.
Note: By default, the results in the aggregated report contain the response that the system has determined to be correct, based on contributor agreement weighted by trust. See the How to Calculate a Confidence Score article to learn more about aggregation. A confidence score is included with every field in the form. This numerical value indicates how confident you can feel that the result is correct.
This tutorial is the simplest way to successfully run a Sentiment Analysis job on the CrowdFlower Platform. To learn more about how to build a job on the Platform, please visit the Success Center.
This tutorial only highlights the basic features provided by the CrowdFlower Basic product. For more information on the features available to you, please visit our Success Center or email us directly at firstname.lastname@example.org. If you don't have an account, you can sign up now.