Crowdsourcing, in which people are engaged to solve tasks online, has revolutionised the way some types of work can be done. From image labelling to text translation and the realisation of personalised medicine, the decisions contributed by the crowd will not always be the same. So can the crowd also be used to correct these decisions?
Crowdsourcing, in which people are engaged to solve tasks online, has revolutionised the way some types of work can be done. From image labelling to text translation, the decisions contributed by the crowd will not always be the same and therefore will need checking.
In their research article from the journal Information Technology, the authors investigate whether the crowd can be used to perform these corrections. This validation process is investigated based on four metrics: the quality of the overall decision making of the crowd; the cost in terms of work done per person; the speed at which the task gets completed and amount of incorrect decisions that are collected.
A game of detectives
The data used in this study were collected using an online game called Phrase Detectives, in which the players were asked to create links between related phrases. For example in the text “Jon went to the shops and he purchased an apple.” The word “he” would be linked to the word “Jon” as mentions of the same entity (in this case, the person Jon).
This linguistic task is called anaphoric co-reference. The game asks the players to create the links and also to validate links created by other players by agreeing or disagreeing with them. Through simulations of game parameters the evaluation shows that an agreement validation stage in the workflow results in fewer decisions required per task, data collection time is decreased and the overall quality of crowd decision making was higher.
More intuitive and enjoyable for users
“Language can be very ambiguous, but it is intuitive for humans to understand it so our research uses crowdsourcing to collect language data to train machine learning algorithms,” explains Chamberlain. “We realised we could not only collect data from the crowd, but also have that data corrected by the crowd, which not only made the system more efficient but also more intuitive and enjoyable for the users.”
This result of this research has implications for the design of future collaborative systems. The study has shown that by allowing groups of people to self-moderate in simple ways (for example through a simple “like” or “upvote” function) the overall decision making from the crowd is improved.
Community Question Answering systems (e.g. StackOverflow or Yahoo! Answers) and social networks (e.g. Facebook) already use this concept on a large scale, although the latter is not typically considered a collaborative working platform.
A better understanding of how people can work together online will facilitate more intuitive design for capturing human knowledge and problem solving.
Read the original article here.
