1,721,018 research outputs found
Aggregation Techniques in Crowdsourcing: Multiple Choice Questions and beyond
No full textCrowdsourcing has been leveraged in various tasks and applications, primarily to gather information from human annotators in exchange for a monetary reward. The main challenge associated with crowdsourcing is the low quality of the results, which can stem from multiple reasons, including bias, error, and adversarial behavior. Researchers and practitioners can apply quality control methods to prevent and detect low-quality responses. For example, worker selection methods utilize qualifications and attention check questions before assigning a task. Similarly, task routing identifies the workers who can provide a more accurate response to a given task type using recommender system techniques. In practice, posterior quality control methods are the most common approach to deal with noisy labels once they are obtained. Such methods require task repetition, i.e., assigning the task to multiple crowd-workers, followed by an aggregation mechanism (aka truth inference) to select the most likely answer or request an additional label. A large number of techniques have been proposed for crowdsourcing aggregation covering several types of task types. This tutorial aims to present common and recent label aggregation techniques for multiple-choice questions, multi-class labels, ratings, pairwise comparison, and image/text annotation. We believe that the audience will benefit from the focus on this specific research area to learn about the best techniques to apply in their crowdsourcing projects
Internet of trees: A vision for advanced monitoring of crops
No full textEcosystem preservation and production maximisation are competing objectives in agriculture. Reducing the need of undifferentiated or late interventions on the crops would reduce the number of disease treatments needed, as well as the consumption of water and fertiliser. This objective is only attainable through crop monitoring systems able to reach a single plant. Precision agriculture employ continuous and pervasive monitoring of crops, that in turn allows fast and targeted interventions. The aim of this paper is to highlight the problems that can be found in designing a wireless sensor network (WSN) able to measure environmental parameters such as relative humidity, irradiance and volatile pollutant concentration and introduces a possible solution that we named the Internet of Trees
CrowdCO-OP : sharing risks and rewards in crowdsourcing
Full text linkPaid micro-task crowdsourcing has gained in popularity partly due to the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a piecework approach to rewards, meaning that workers are paid for each task they complete, given that their work quality is considered sufficient by the requester or the platform. Such an approach creates risks for workers; their work may be rejected without being rewarded, and they may be working on poorly rewarded tasks, in light of the disproportionate time required to complete them. As a result, recent research has shown that crowd workers may tend to choose specific, simple, and familiar tasks and avoid new requesters to manage these risks. In this paper, we propose a novel crowdsourcing reward mechanism that allows workers to share these risks and achieve a standardized hourly wage equal for all participating workers. Reward-focused workers can thereby take up challenging and complex HITs without bearing the financial risk of not being rewarded for completed work. We experimentally compare different crowd reward schemes and observe their impact on worker performance and satisfaction. Our results show that 1) workers clearly perceive the benefits of the proposed reward scheme, 2) work effectiveness and efficiency are not impacted as compared to those of the piecework scheme, and 3) the presence of slow workers is limited and does not disrupt the proposed cooperation-based approaches
Fair virtualization of 802.11 networks
No full textWe consider virtualization of network capacity in 802.11 WLANs and mesh networks. We show that allocating total airtime slices to ISPs is analogous to allocating a fraction of available time-slots in TDMA. We establish that the max-min fair flow rate allocation within an ISP airtime slice can be characterized independently of the rate allocation policy employed in other slices. Building on these observations, we present a lightweight, distributed algorithm for allocating airtime slices among ISP and max-min fair flow rates within each slice
Fast, responsive decentralized graph coloring
Full text linkGraph coloring problem arises in numerous networking applications. We solve it in a fully decentralized way (ı.e., with no message passing). We propose a novel algorithm that is automatically responsive to topology changes, and we prove that it converges to a proper coloring in O(NlogN) time with high probability for generic graphs, when the number of available colors is greater than Δ , the maximum degree of the graph, and in O(logN) time if Δ=O(1) . We believe the proof techniques used in this paper are of independent interest and provide new insight into the properties required to ensure fast convergence of decentralized algorithms
Worker Perspectives on Designs for a Crowdwork Co-operative
No full textCrowdwork platforms such as Amazon Mechanical Turk (AMT) are a crucial infrastructural component of our global data assemblage. Through these platforms, low-paid crowdworkers perform the vital labour of manually labelling large-scale and complex datasets, labels that are needed to train machine learning and AI models (Tubaro et al., Big Data & Society, 7(1), 2020) and which enable the functioning of much digital technology, from niche applications to global platforms such as Google, Amazon and Facebook. In this chapter, we reflect on how a ‘design justice’ approach might be valuable to build on insights gained from a series of exploratory discussions we have engaged in with US-based crowdworkers about how a crowdworker co-operative might work in practice, and begin to sketch out a potential software architecture that could form the basis of future participative approaches to the design and development of a crowdworker co-operative. We begin by describing and reflecting on our own evolving methodology and how it fits with the ‘design justice’ lens we propose for future work. Following this, we present findings from our discussions with crowdworkers about how a crowdwork co-operative might work in practice, including what values workers would like to see embedded in the design. We then finish with the outline of a prototype software architecture for a crowdworker co-operative that could be used as a starting point in future design work in collaboration with crowdworkers
Addressing labour exploitation in the data science pipeline: views of precarious US-based crowdworkers on adversarial and co-operative interventions
No full textPurpose: Underlying much recent development in data science and artificial intelligence (AI) is a dependence on the labour of precarious crowdworkers via platforms such as Amazon Mechanical Turk. These platforms have been widely critiqued for their exploitative labour relations, and over recent years, there have been various efforts by academic researchers to develop interventions aimed at improving labour conditions. The aim of this paper is to explore US-based crowdworkers’ views on two proposed interventions: a browser plugin that detects automated quality control “Gold Question” (GQ) checks and a proposal for a crowdworker co-operative. Design/methodology/approach: The authors interviewed 20 US-based crowdworkers and undertook a thematic analysis of collected data. Findings: The findings indicate that US-based crowdworkers tend to have negative and mixed feelings about the GQ detector, but were more enthusiastic about the crowdworker co-operative. Originality/value: Drawing on theories of precarious labour, this study suggests an explanation for the findings based on US-based workers’ objective and subjective experiences of precarity. The authors argue that for US-based crowdworkers “constructive” interventions such as a crowdworker co-operative have more potential to improve labour conditions
BLC: Private Matrix Factorization Recommenders via Automatic Group Learning
Full text linkWe propose a privacy-enhanced matrix factorization recommender that exploits the fact that users can often
be grouped together by interest. This allows a form of “hiding in the crowd” privacy. We introduce a novel
matrix factorization approach suited to making recommendations in a shared group (or “nym”) setting and
the BLC algorithm for carrying out this matrix factorization in a privacy-enhanced manner. We demonstrate
that the increased privacy does not come at the cost of reduced recommendation accuracy
The Unintended Consequences of Automated Scripts in Crowdwork Platforms: A Simulation Study in MTurk
No full textCrowdworkers on platforms like Amazon Mechanical Turk face growing competition as a result of the global excess supply of digital labour. As a result, many crowdworkers turn to automated scripts, which help them locate better tasks faster and to boost their earnings. However, to date, it is not clear whether and to what extent the use of such scripts influence the opportunities for those crowdworkers who do not use them. This an important aspect that warrants further exploration because it can have negative implications for the health of crowdwork platforms. In this study, we use Discrete Event Simulation to identify and quantify the unintended consequences of the excessive use of automated scripts. Our findings show that, while the use of scripts allows some crowdworkers to identify and accept far more tasks than others, in the long run, this behaviour results in their competence persistence and reputational persistence and progressively to detrimental impacts for those workers who do not use scripts, and who may ultimately be forced to exit the platform. As a result, automated scripts have negative consequences, whereby their excessive use leads to a tragedy of the commons for all platform stakeholders, including the crowdworkers, the job requesters and the platform itself
- …
