Once you have built a job, added data, configured settings, and proceeded to launch, you will notice that CrowdFlower's system presents you with an estimation of how much your job will cost. This estimate is based on the settings that you have selected, as well as on the volume of data you have uploaded to be processed in your job. Upon launching, you may also notice that the job's cost differs somewhat from the initial estimate.
We have honed our cost estimation system to give you a highly accurate estimation and reserve the proper amount of funds to allow you to run your job uninterrupted. However, certain contingencies that occur during a job's lifespan may cause the job's cost to diverge from this initial estimate. In the following article, we will explain how your initial job cost is estimated, why actual cost may differ from this initial estimate, what job costs you pay for, and what you can do to help keep your job cost low.
Here are some questions that are answered in this article:
- How is my initial job cost estimated?
- Why does my job cost change from the original estimate while the job is running?
- What costs do I pay for?
- Why do I pay for untrusted work?
- What are the best ways to keep my job costs down?
How is my initial job cost estimated?
The individual cost of an Assignment
Your CrowdFlower Transaction Fee
The estimated number of Gold /Test Question judgments (including untrusted judgments in work mode) based on the volume of data you have uploaded and your judgment settings
The number of test questions shown per assignment
Variable judgments mode (To be safe, our system assumes that units will collect the maximum number of variable judgments.)
Here’s the basic formula used to calculate the estimated job cost:
(Judgments per row * (Pages of work * Price per page)) + buffer + transaction fee = estimated job cost
Say you’re launching a job with 1,000 rows of data and you choose to display 10 rows per page; that actually means that one of those rows is a test question, while the other 9 are rows of data. This yields about 112 pages of work for the job. You decide to collect 3 judgments per row and set your payment to 10 cents per page. The basic cost estimate, then, is:
3* (112 * $0.10) = $33.60
Now we have to think about the transaction fee. If you’re paying a 20% transaction fee, that means you’re actually paying 12 cents per page of work, so the estimate would look like this:
3 * (112 * $0.12) = $40.32
This is where it gets a bit more complicated. The buffer is determined by a machine learning algorithm that takes data from previous jobs run on our platform with similar settings. The buffer is essentially the amount we expect you to pay for untrusted judgments that your job collects; this is predicted based on how many untrusted judgments other jobs of yours have collected. Of course, this is just an estimate - perhaps this time around, fewer contributors’ judgments will become untrusted. In any case, maybe a previous job that had similar settings collected 100 untrusted judgments, so about 11 pages of untrusted work. That would be about 11 * $0.12, so about $1.32 extra. Now your estimate may resemble the following:
(3 * (112 * $0.12)) + $1.32 = $41.64
This estimate can change as the job runs. Continue reading for further information.
Why does my job cost change from the original estimate while the job is running?
The collection of more untrusted work in Work mode than expected
Any increase to Payment Cents per Assignment that you make
The cost of untrusted work resulting from manual rejections of contributors that you make
What costs do I pay for?
The cost of all trusted and untrusted judgments in Work mode
The cost of all trusted and untrusted judgments on Test Questions in Work Mode
Your CrowdFlower Transaction Fee
Why do I pay for untrusted judgments?
What are the best ways to keep my job costs down?
Make sure that your Test Questions are fair and correct. Bad Test Questions are a major source of avoidable untrusted work: they cause high-quality contributors to be removed from your job, thereby causing their work to become untrusted.
Monitor your test questions. Sometimes bad Test Questions are hard to avoid at the time of launch. You can always review and update them while a job is live. Doing so will decrease untrusted work as it increases throughput and contributor satisfaction.