1,720,965 research outputs found
On the Role of Task Design in Crowdsourcing Campaigns
No full textDespite the success of crowdsourcing marketplaces, fully harnessing their massive workforce remains challenging. In this work we study the effect on crowdsourcing campaigns of different feedback and payment strategies. Our results reveal the joint effect of feedback and payment on the quality and quantity of the outcome
A workload-dependent task assignment policy for crowdsourcing
No full textCrowdsourcing marketplaces have emerged as an effective tool for high-speed, low-cost labeling of massive data sets. Since the labeling accuracy can greatly vary from worker to worker, we are faced with the problem of assigning labeling tasks to workers so as to maximize the accuracy associated with their answers. In this work, we study the problem of assigning workers to tasks under the assumption that workers' reliability could change depending on their workload, as a result of, e.g., fatigue and learning. We offer empirical evidence of the existence of a workload-dependent accuracy variation among workers, and propose solution procedures for our Crowdsourced Labeling Task Assignment Problem, which we validate on both synthetic and real data sets
The dimensions of crowdsourcing task design
No full textCrowdsourcing, i.e., the provision of micro-tasks to be executed by a large pool of possibly anonymous workers, is attracting an increasing research attention, because it promises to help solving many scientific and practical problems where the harmonic cooperation of humans and machines delivers superior results. This paper proposes a systematic view of the crowdsourcing task design space and categorizes the dimensions that qualify the design decisions in crowdsourcing applications. For each dimension, we discuss the main open research problems and the most significant contributions, thereby offering guidelines for a principled understanding of current crowdsourcing marketplaces
Towards an unbiased approach for the evaluation of social data geolocation
Get PDFWe present a study that reveals a significant statistical bias in the distributions of geolocated and non-geolocated social data. We state that this bias affects the real performance of social geolocation algorithms and can impair the results of these algorithms, which are commonly trained and tested on datasets consisting of crawled geolocated data. At last, we propose the construction of an a-posteriori geolocated dataset for an unbiased estimation of new and state-of-the-art algorithms alike
Champagne: A Web Tool for the Execution of Crowdsourcing Campaigns
No full textWe present Champagne, a web tool for the execution of crowdsourcing campaigns. Through Champagne, task requesters can model crowdsourcing campaigns as a sequence of choices regarding different, independent crowdsourcing design decisions. Such decisions include, e.g., the possibility of qualifying some workers as expert reviewers, or of combining different quality assurance techniques to be used during campaign execution. In this regard, a walkthrough example showcasing the capabilities of the platform is reported. Moreover, we show that our modular approach in the design of campaigns overcomes many of the limitations exposed by the major platforms available in the market
Top-k diversity queries over bounded regions
No full textTop-k diversity queries over objects embedded in a low-dimensional vector space aim to retrieve the best k objects that are both relevant to given user’s criteria and well distributed over a designated region. An interesting case is provided by spatial Web objects, which are produced in great quantity by location-based services that let users attach content to places and are found also in domains like trip planning, news analysis, and real estate. In this paper we present a technique for addressing such queries that, unlike existing methods for diversified top-k queries, does not require accessing and scanning all relevant objects in order to find the best k results. Our Space Partitioning and Probing (SPP) algorithm works by progressively exploring the vector space, while keeping track of the already seen objects and of their relevance and position. The goal is to provide a good quality result set in terms of both relevance and diversity. We assess quality by using as a baseline the result set computed by MMR, one of the most popular diversification algorithms, while minimizing the number of accessed objects. In order to do so, SPP exploits score-based and distance-based access methods, which are available, e.g., in most geo-referenced Web data sources. Experiments with both synthetic and real data show that SPP produces results that are relevant and spatially well distributed, while significantly reducing the number of accessed objects and incurring a very low computational overhead
Improving Health Monitoring With Adaptive Data Movement in Fog Computing
Get PDFPervasive sensing is increasing our ability to monitor the status of patients not only when they are hospitalized but also during home recovery. As a result, lots of data are collected and are available for multiple purposes. If operations can take advantage of timely and detailed data, the huge amount of data collected can also be useful for analytics. However, these data may be unusable for two reasons: data quality and performance problems. First, if the quality of the collected values is low, the processing activities could produce insignificant results. Second, if the system does not guarantee adequate performance, the results may not be delivered at the right time. The goal of this document is to propose a data utility model that considers the impact of the quality of the data sources (e.g., collected data, biographical data, and clinical history) on the expected results and allows for improvement of the performance through utility-driven data management in a Fog environment. Regarding data quality, our approach aims to consider it as a context-dependent problem: a given dataset can be considered useful for one application and inadequate for another application. For this reason, we suggest a context-dependent quality assessment considering dimensions such as accuracy, completeness, consistency, and timeliness, and we argue that different applications have different quality requirements to consider. The management of data in Fog computing also requires particular attention to quality of service requirements. For this reason, we include QoS aspects in the data utility model, such as availability, response time, and latency. Based on the proposed data utility model, we present an approach based on a goal model capable of identifying when one or more dimensions of quality of service or data quality are violated and of suggesting which is the best action to be taken to address this violation. The proposed approach is evaluated with a real and appropriately anonymized dataset, obtained as part of the experimental procedure of a research project in which a device with a set of sensors (inertial, temperature, humidity, and light sensors) is used to collect motion and environmental data associated with the daily physical activities of healthy young volunteers
Achieving quality in crowdsourcing through task design and assignment
No full textIl crowdsourcing, ossia la pratica di affidare la realizzazione di un task ad un insieme indefinito di persone (crowd), ha il potenziale per rivoluzionare il modo in cui le persone lavorano su web. L’utilizzo del crowdsourcing permette alle aziende di raccogliere e aggregare contributi in maniera totalmente distribuita, e ai lavoratori (crowd worker) di guadagnare senza la necessità di un luogo di lavoro fisico o di una particolare forma contrattuale. Grazie a tale flessibilità, il crowdsourcing si sta affermando sia nel settore industriale che in quello accademico, ed è stato stimato che, a livello globale, le diverse aziende abbiano il potenziale per esternalizzare task in crowdsourcing per un valore complessivo di 300 miliardi di dollari. Con l’aumento del numero di aziende che decidono di utilizzare il crowdsourcing, i crowd worker rischiano però di diventare una risorsa limitata. Una questione importante è dunque quella di comprendere come riuscire ad ottenere, preservare e persuaduare la crowd a contribuire. In particolare, questo lavoro di tesi si focalizza sulla studio di quali meccanismi siano efficaci per far sì che la crowd fornisca contributi di alta qualità. L’approccio seguito è duplice. Da un parte, ci si concentra su come una attenta progettazione dei task possa aiutare a migliorare la qualità dei risultati. In tal senso, viene presentata una caratterizzazione dello spazio di design dei task di crowdsourcing, che viene poi messa a confronto con le capacità delle piattaforme attualmente presenti nel mercato. Dall’altra parte, si pone l’attenzione sul problema di come assegnare i task ai crowd worker. A questo proposito, l’ipotesi di una diversa accuratezza tra i lavoratori si è già rivelata essere un’assunzione fondamentale per riuscire ad aumentare la qualità dei risultati. Alla luce di ciò, viene proposta una politica di assegnamento dei task che tenga in conto dei diversi livelli di abilità dei lavoratori, sotto l’ipotesi che questi possano esibire valori di accuratezza varibili e dipendenti dal loro attuale carico di lavoro. Un’ampia sezione sperimentale va a supportare tali supposizioni. In particolare, sono stati condotti esperimenti al fine di identificare quali fattori della progettazione di un task possano influenzare la qualità del risultato. Inoltre, si è verificata l’esistenza di un fenomeno di affaticamento/apprendimento tra i lavoratori. In ultimo, la politica di assegnamento proposta è stata estensivamente validata utilizzando dati sia sinteticamente generati che provenienti da crowd reali.Crowdsourcing, i.e., the assembling of strangers to accomplish a task, has the potential to revolutionize the way people work on the web. The promotion of crowdsourcing initiatives allows companies to easily collect and compound contributions in a distributed fashion, while letting individuals work and earn without the need for a physical working place or pre-existing employment contracts. Thanks to its high flexibility, crowdsourcing is gain- ing a more prominent role in both the industry and academia, and it has been estimated that companies have the potential to crowdsource more than 300 billion USD of work worldwide. As the number of organizations em- bracing crowdsourcing is increasing, crowd workers are likely to become a limited resource. An important issue is therefore to understand how to obtain, retain and persuade a crowd to contribute. In this work, we are es- pecially interested in understanding which mechanisms are effective for eliciting high quality contributions from the crowd. Our approach is twofold. On the one hand, we focus on how a careful task design can help improve quality of contributions. We present a characterization of the design space of crowdsourcing tasks, and we then contrast the capabilities offered by the commercially-available platforms against the proposed characterization. On the other hand, we turn our attention to the problem of assigning tasks to crowd workers. In this respect, carefully considering workers’ accuracy has already proved to be the key enabler for increasing task quality. We therefore propose a task assignment policy to support the assignment of tasks in relation to crowd workers’ abilities, under the assumption that workers may exhibit varying accuracy depending on their workload. We validate our findings through an extensive experimental phase. Specifically, we conduct experiments with the aim of verifying which task design dimensions affect the quality of the outcome. Moreover, we offer empirical evidence of the existence of a fatigue/learning phenomenon among workers, and we extensively validate the proposed task assignment procedure against both synthetic and real data.DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIAComputer Science and Engineering27PERNICI, BARBARAFIORINI, CARLO ETTOR
- …
