Follow

Tips for Job Design

As you begin to design your first job, keep in mind that careful planning pays off. Creating wireframes, writing focused, succinct, unambiguous questions, and thinking about your rows from the point of view of your contributors will go a long way toward launching a successful job.

Before You Start

Prior to creating a job, make sure you can articulate the following:

  • The problem you are trying to solve
  • An adequate solution to that problem
  • The desired form of output or data you hope to collect
  • The characteristics of your job that make it suitable for crowdsourcing 

If you can confidently speak to each of these items, you’re ready to begin the planning phase with job wireframing.

Understand What You're Trying to Achieve

Successful crowdsourcing requires a systematic approach to job design. Before creating your job's User Interface, you’ll need a clear understanding of the optimal workflow for completing your job. Ask yourself:

  • What questions will you ask the crowd? In what order will you ask them?
  • How will you ask these questions? How can you eliminate any ambiguity?
  • How will the crowd provide answers? (e.g. drop-down menu, radio buttons, text field, etc.)
  • Would your job's UI benefit from logic or contingencies?

Diagramming your job workflow as a table or decision tree is an important first step to meeting your job objectives.

Write Clear and Concise Questions

It’s surprisingly easy to get carried away by creating long-winded questions. We urge you to keep them short. Long forms with many questions can slow down and confuse contributors. In cases where your job's User Interface exceeds 8 questions, we advise you to break your job into several smaller jobs.

Newspaper editors instruct journalists to write at a fifth grade level. We’ve found it pays to follow this rule of thumb. Contributors come from many linguistic and cultural backgrounds, so clear and concise language is critical for contributors to do their jobs efficiently and effectively.

Additional helpful tips:

  • Provide any prerequisite background information that could help contributors complete jobs
  • In your instructions, explain how you will use the results
  • Double check instructions for clarity and accuracy by having a colleague or acquaintance review them as though they were a contributor
  • Review the language in your questions to ensure there is no bias toward a particular answer

Consider User Experience

  • Minimize scrolling & clicking: CML forms with lots of moving parts like scroll bars and buttons can become confusing. Carefully consider your UI layout to reduce contributor fatigue and improve efficiency.
  • Keep it local: Whenever possible, avoid asking contributors to navigate away to external sites to complete a job.
  • Provide shortcuts and hyperlinks: If your job requires navigating to other webpages or searching the Internet, make sure you support contributors with shortcuts, such as hyperlinks to a predefined google search in the following format: http://www.google.com/search?q=some+query. When using our custom Liquid validator you can use the following format to encode an unknown piece of data: http://www.google.com/search?q={{ your_data | urlencode }}. You can also create a link with an HTML tag in the following way: <a href="http://www.google.com/search?q=your_query" target="_blank" />.
  • View your job through the contributor’s eyes: Use CrowdFlower’s Preview tab to see how the first few rows of data are populated into your UI. The preview page is a great way to test any logic and experience your UI as contributors will.

Consider Data Collection

  • Avoid text fields: Text fields provide data that is challenging to normalize and aggregate. When possible, use deterministic UI elements like dropdowns, radio buttons, and checkboxes rather than text fields. 
  • Use validation: When using text fields is unavoidable, we recommend using a validator to normalize text entries. CrowdFlower offers validators for phone numbers, addresses, integers, proper case, and many other types of responses. For more information, see our article on Validations.
  • Use name and label attributes: Using name as well as label attributes helps to avoid complications that can arise when generating reports down the road. For example, instead of <cml:checkboxes label="Check one of the following"> use <cml:checkboxes label="Check one of the following" name="my_burrito_preferences">. For any attributes, avoid using special characters or letters with accents in attribute fields.

Graphical Editor vs. Code Editor

Once you've settled on a workflow for your job, it’s time to build a User Interface for your job. There are two form editors you can use to create your interface: The Graphical Editor and the Code Editor.

  • The Graphical Editor is the ideal tool for creating a simple UI.
  • The Code Editor allows you to use HTML, Javascript, and CSS to create and customize your UI. Using the Code Editor is the only way to implement multiple layer logic into your UI. If you choose to use the Code Editor, we recommend first reviewing the CML Documentation.

Note: Some users have found success setting up basic UIs in the Graphical Editor before switching to the Code Editor to add more complex features. Because the Code Editor supports much more sophisticated HTML/CML/JS that is not compatible with the Graphical Editor. Because of this, once saving in the Code Editor you will not be able to return to using the Graphical Editor.

You’re On Your Way!

With careful consideration of your job’s flow, your contributors’ needs, and your data collection demands, your first steps into the world of crowdsourcing are sure to be a success. If you’re new to this realm, keep in mind that there’s always room for improvement as you gain more experience with this type of job design. Things rarely go precisely as planned when you’re just starting out, which is a great reason to carry out some usability testing before you push your first job live. Do this by launching your job on the CrowdFlower Internal Channel and accessing your units from the link located on the bottom of the job's home page. Finally, remember to follow up on your job’s test questions. Tracking which test questions are missed can highlight problem areas and provide insight into possibilities for future iterations.

From the folks at CrowdFlower, good luck!


Was this article helpful?
2 out of 3 found this helpful


Have more questions? Submit a request
Powered by Zendesk