In addition to Internal or External (General) channel selection, there is additional option known as Custom Channels. This selection allows the launching of jobs to groups of preferred contributors created by the requestor's team. This feature is currently available as an invite-only Beta for customers with a paid subscription. Contact your Customer Success Manager for more details.
Custom Channels improves judgement quality by only launching a job to contributors who have demonstrated specific skills when working on similar jobs or data in the past. This results in:
- Lower crowd cost by reducing the number of untrusted judgements from unskilled contributors.
- Ability to run more complicated/advanced CrowdFlower jobs that require contributors with a particular skill set
Custom Channels can be created on the Contributor page (located in Settings) on any job.
Channels created this way are available to all team members to be used on any team job.
Accuracy Based Custom Channels
Accuracy based custom channels can be used to create a group of preferred contributors from a previously run job. To do so, select one or more finished jobs and an accuracy threshold for each job. In the absence of qualifying jobs, you can also run a specialized job, similar to the target job, to create a Custom Channel.
CrowdFlower automatically evaluates contributors that meet or exceed the accuracy threshold you set to identify qualifying contributors and associate them with the named Custom Channel. When multiple jobs are chosen to source a Custom Channel, any contributor who meets accuracy threshold in any of the jobs qualifies to be in the channel.
Note: When Creating Custom Channels throughput can be impacted (especially for jobs that were previously targeted to leveled crowd) but as more contributors are chosen, throughput should increase.
Manual Custom Channels
Manual Custom Channels can be created using a list of specific preferred contributors who have been determined qualified to work in a specific job. This channel can be created by uploading a CSV file with contributor IDs.
Note: The CSV must contain exactly one column with contributor IDs and no column headers.
All contributors, their ID, overall accuracy, number of judgments and additional information can be found in the job Monitor > Contributors page. Performant contributors can also be found from reviewing the Contributor or Full report in a job.
How to Update a Custom Channel
Accuracy based Custom Channels can be updated by adding or deleting specified jobs and by modifying the accuracy threshold associated with each job. When a Custom Channel is updated, it is regenerated by clearing the previous list of contributors and adding contributors who qualify according to the new criteria.
An existing manual Custom Channel can be updated by uploading a new CSV file. The new set of contributors replaces the existing set. If you want to add contributors to a manual Custom Channel, upload a file with both the existing and new contributor IDs. This can take a few minutes to update.
Launching jobs to Custom Channels
Jobs can be launched to one or more custom channels of any type using the Contributor settings page. These jobs are exclusive to custom channels and cannot be launched to other contributor in the General External Crowd. The job may still be worked on via the Internal Channel. If multiple custom channels are selected, a contributor who is part of at least one of them can work on the job.
To get best results, always make sure to generate custom channels from similar jobs and use cases as Custom Channels override other skills applied to a job. For example, if a job was originally launched to level 3 contributors with knowledge of French, once a custom channel is applied to the job, the level and language setting is ignored. But if a Custom Channel is generated from a job with similar settings, the contributors will inherently have these skills.
Also, once a custom channel is applied to a job, level, language and geography preferences are removed from the job. If you wish to target the crowd again, you will need to re-apply these settings. Note that, to change channel settings, a job must be paused. On resuming the job, new judgements are collected only from qualifying contributors.
Custom Channels and Test Questions
Custom Channels result is cost savings by reducing the number of untrusted judgments. However, similar to jobs launched to leveled contributors, it is important for jobs launched to Custom Channels to have test questions. While custom channels enable targeting contributors who have demonstrated quality in previous jobs, test questions evaluate judgements from these contributors continuously to ensure ensure that the contributors maintain quality over time.
Contributors who fail to maintain quality are removed from the job similar to jobs targeted to the General External Crowd. Once flagged or banned, contributors in the custom channel are not eligible to work on the job.