Follow

Job Costs FAQ

Once you have built a job, added data, configured settings, and proceeded to launch, you will notice that CrowdFlower's system presents you with an estimation of how much your job will cost.  This estimate is based on the settings that you have selected, as well as on the volume of data you have uploaded to be processed in your job.  Upon launching, you may also notice that the job's cost differs somewhat from the initial estimate.  

We have honed our cost estimation system to give you a highly accurate estimation and reserve the proper amount of funds to allow you to run your job uninterrupted.  However, certain contingencies that occur during a job's lifespan may cause the job's cost to diverge from this initial estimate.  In the following article, we will explain how your initial job cost is estimated, why actual cost may differ from this initial estimate, what job costs you pay for, and what you can do to help keep your job cost low.

Here are some questions that are answered in this article:

How is my initial job cost estimated?

At the time of launch, our system will generate an estimated job cost for you based on the following criteria:
  • The individual cost of an Assignment
  • Your CrowdFlower Transaction Fee
  • The estimated number of Gold /Test Question judgments (including untrusted judgments in work mode) based on the volume of data you have uploaded and your judgment settings
  • The number of test questions shown per assignment
  • Variable judgments mode (To be safe, our system assumes that units will collect the maximum number of variable judgments.)

Here’s the basic formula used to calculate the estimated job cost:

(Judgments per row * (Pages of work * Price per page)) + buffer + transaction fee = estimated job cost

Say you’re launching a job with 1,000 rows of data and you choose to display 10 rows per page; that actually means that one of those rows is a test question, while the other 9 are rows of data. This yields about 112 pages of work for the job. You decide to collect 3 judgments per row and set your payment to 10 cents per page. The basic cost estimate, then, is:

3* (112 * $0.10) = $33.60

Now we have to think about the transaction fee. If you’re paying a 20% transaction fee, that means you’re actually paying 12 cents per page of work, so the estimate would look like this:

3 * (112 * $0.12) = $40.32

This is where it gets a bit more complicated. The buffer is determined by a machine learning algorithm that takes data from previous jobs run on our platform with similar settings. The buffer is essentially the amount we expect you to pay for untrusted judgments that your job collects; this is predicted based on how many untrusted judgments other jobs of yours have collected. Of course, this is just an estimate - perhaps this time around, fewer contributors’ judgments will become untrusted. In any case, maybe a previous job that had similar settings collected 100 untrusted judgments, so about 11 pages of untrusted work. That would be about 11 * $0.12, so about $1.32 extra. Now your estimate may resemble the following:

(3 * (112 * $0.12)) + $1.32 = $41.64

This estimate can change as the job runs. Continue reading for further information.

Note: All jobs presently reserve a minimum of $10 of funds to ensure that they are adequately funded to run to completion.
 

Why does my job cost change from the original estimate while the job is running?

You may occasionally notice that your job costs have increased from the original estimate.  If your account is low on funds, you may even need to add additional funds in order to continue running your job.  The following are all reasons why a job's cost may increase after launch:
  • The collection of more untrusted work in Work mode than expected
  • Any increase to Payment Cents per Assignment that you make
  • The cost of untrusted work resulting from manual rejections of contributors that you make

What costs do I pay for?

Your actual job costs are comprised of the following:
  • The cost of all trusted and untrusted judgments in Work mode
  • The cost of all trusted and untrusted judgments on Test Questions in Work Mode
  • Your CrowdFlower Transaction Fee
 

Why do I pay for untrusted judgments?

Untrusted judgments occur when a contributor's accuracy score falls below the threshold set in the job. Untrusted judgments therefore begin as trusted judgments, which are all paid for as trusted judgments are considered valid work from the contributors. Essentially, any assignment submitted in Work Mode is paid for in the job, so any work prior to the contributor becoming untrusted is paid out.
Untrusted judgments are not included in the aggregated job results, and rows that have collected untrusted judgments will be rereleased to the crowd to recollect trusted judgments. New judgments will accrue additional payments, and therefore, the job cost will increase. To prevent this from happening on a large scale, monitor test questions closely and make sure the accuracy threshold is set reasonably.

 

What are the best ways to keep my job costs down?

Running a job smoothly and at a low cost is all about preparation: the job settings and quality control measures that CrowdFlower offers will help you keep your costs low while collecting top-quality data.  Here are a few things you can do before launch and during a job's lifespan to help keep costs down:
  • Make sure that your Test Questions are fair and correct.  Bad Test Questions are a major source of avoidable untrusted work: they cause high-quality contributors to be removed from your job, thereby causing their work to become untrusted.
  • Monitor your test questions.  Sometimes bad Test Questions are hard to avoid at the time of launch.  You can always review and update them while a job is live.  Doing so will decrease untrusted work as it increases throughput and contributor satisfaction. 

Was this article helpful?
3 out of 3 found this helpful


Have more questions? Submit a request
Powered by Zendesk