790 research outputs found
SASWeb 2012: Semantic and Adaptive Social Web
No full textSASWeb 2012: Semantic and Adaptive Social Web
organized by Lora Aroyo, Federica Cena, Antonina Dattolo, Pasquale Lops, Julita Vassileva
(1) Building multi-layer social knowledge maps with Google Maps API
MinEr Liang, Julio Guerra, Peter Brusilovsky
(2) Learning from a network of peers via peer-driven adjustment of a corpus
John Champaign, Robin Cohen
****Invited Talks
(4) Culture in User Modeling 3.0
Jacqueline Bourdeau
(5) Leveraging social and semantic components in adaptive environments
Cristina Gena
(6) Meaning is its use: towards the use of distributional semantics for content-based recommender systems
Cataldo Musto
(7) Exploring folksonomies for adaptive query expansion
Fabio Gasparett
30 sentences annotated by 15 crowd workers (3/3)
No full text<p>These are 30 sentences annotated by 15 crowd workers each, within the context of the project Crowd Watson (http://crowd-watson.nl) for medical relation extraction.</p>
<p>Project members: Chris Welty (IBM Research), Lora Aroyo (VU University Amsterdam),</p
30 sentences annotated by 15 crowd workers (2/3)
No full text<p>These are 30 sentences annotated by 15 crowd workers each, within the context of the project Crowd Watson (http://crowd-watson.nl) for medical relation extraction.</p>
<p>Project members: Chris Welty (IBM Research), Lora Aroyo (VU University Amsterdam),</p
30 sentences annotated by 15 crowd workers (1/3)
No full text<p>These are 30 sentences annotated by 15 crowd workers each, within the context of the project Crowd Watson (http://crowd-watson.nl) for medical relation extraction. </p>
<p> </p>
<p>Project members: Chris Welty (IBM Research), Lora Aroyo (VU University Amsterdam), </p
First International Workshop on User Interfaces for Crowdsourcing and Human Computation
No full textRecent years witnessed an explosion in the number and variety of data crowdsourcing initiatives. From OpenStreetMap to Amazon Mechanical Turk, developers and practitioners have been striving to create user interfaces able to effectively and efficiently support the creation, exploration, and analysis of crowdsourced information. The extensive usage of crowdsourcing techniques brings a major change of paradigm with respect to traditional user interface for data collection and exploration, as effectiveness, speed, and interaction quality concerns play a central role in supporting very demanding incentives, including monetary ones. The First International Workshop on User Interfaces for Crowdsourcing and Human Computation (CrowdUI 2014), co-located with the AVI 2014 conference, brought together researchers and practitioners from a wide range of areas interested in discussing the user interaction challenges posed by crowdsourcing systems. © 2014 ACM
FrameNet Semantic Frame Disambiguation with CrowdTruth
No full text<p>This repository contains a ground truth corpus for semantic frame disambiguation, acquired with crowdsourcing and processed with <strong><a href="http://crowdtruth.org/">CrowdTruth</a></strong> metrics that capture ambiguity in annotations by measuring inter-annotator disagreement.</p>
<p>The dataset contains annotations for 433 sentence-word pairs from the <a href="https://framenet.icsi.berkeley.edu/">FrameNet corpus v.1.7</a>, with each sentence-word pair annotated for frame disambiguation by 15 workers. The crowdsourced data was collected from <a href="https://www.mturk.com/">Amazon Mechanical Turk</a>.</p>
<p>The corpus has been referenced in the following paper:</p>
<ul>
<li>Anca Dumitrache, Lora Aroyo and Chris Welty: <strong><a href="https://arxiv.org/abs/1805.00270">Capturing and Interpreting Ambiguity in Crowdsourcing Frame Disambiguation</a></strong>. <a href="https://www.humancomputation.com/2018/">HCOMP 2018</a>.</li>
</ul>
<p>To replicate the data processing from the paper, use the Jupyter Notebook file <code>CrowdTruth metrics.ipynb</code>. It requires the installation of the <a href="https://github.com/CrowdTruth/CrowdTruth-core">CrowdTruth metrics</a> Python package (v >= 2.0).</p>
<p>The data aggregated with CrowdTruth metrics is available in folder <code>data/output/</code></p>
<p>The raw crowdsourcing data is available in folder <code>data/input/</code></p>
<p>If you find this data useful in your research, please consider citing:</p>
<pre><code>@inproceedings{dumitrache2018frames,
Author = {Anca Dumitrache and Lora Aroyo and Chris Welty},
Title = {Capturing Ambiguity in Crowdsourcing Frame Disambiguation},
Booktitle = {The sixth AAAI Conference on Human Computation and Crowdsourcing},
Year = {2018}
}
</code></pre>
CrowdTruth 2.0:Quality metrics for crowdsourcing with disagreement
No full textTypically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. In this paper, we present ongoing work into the CrowdTruth metrics, that capture and interpret inter-annotator disagreement in crowdsourcing. The CrowdTruth metrics model the inter-dependency between the three main components of a crowdsourcing system – worker, input data, and annotation. The goal of the metrics is to capture the degree of ambiguity in each of these three components. The metrics are available online at https://github.com/CrowdTruth/CrowdTruth-core.</p
Validation Methodology for Expert-Annotated Datasets: Event Annotation Case Study
Full text linkEvent detection is still a difficult task due to the complexity and the ambiguity of such entities. On the one hand, we observe a low inter-annotator agreement among experts when annotating events, disregarding the multitude of existing annotation guidelines and their numerous revisions. On the other hand, event extraction systems have a lower measured performance in terms of F1-score compared to other types of entities such as people or locations. In this paper we study the consistency and completeness of expert-annotated datasets for events and time expressions. We propose a data-agnostic validation methodology of such datasets in terms of consistency and completeness. Furthermore, we combine the power of crowds and machines to correct and extend expert-annotated datasets of events. We show the benefit of using crowd-annotated events to train and evaluate a state-of-the-art event extraction system. Our results show that the crowd-annotated events increase the performance of the system by at least 5.3%
Nichesourcing for Improving Access to Linked Cultural Heritage Datasets
Full text linkSchreiber, A.T. [Promotor]Aroyo, L.M. [Promotor]Boer, V. de [Copromotor
CrowdTruth Corpus for Open Domain Relation Extraction from Sentences
No full text<p>This repository contains a ground truth corpus for open domain relation extraction from sentences, acquired with crowdsourcing and processed with <strong><a href="http://crowdtruth.org/">CrowdTruth</a></strong> metrics that capture ambiguity in annotations by measuring inter-annotator disagreement.</p>
<p>The dataset contains annotations for 4,100 sentences sampled from Angeli et al. (1) and Riedel et al. (2), over 16 relations, with each sentence annotated by 15 workers. The sentences have been pre-processed with Distant Supervision (3) using the Freebase knowledge base, in order to identify the term pairs in each sentence that are likely to express a relation. The crowdsourced data was collected from <a href="http://figure-eight.com/">Figure Eight</a> and <a href="https://www.mturk.com/">Amazon Mechanical Turk</a>.</p>
<p>This corpus has been discussed in the following papers:</p>
<ul>
<li>Anca Dumitrache, Lora Aroyo and Chris Welty: <strong><a href="https://arxiv.org/abs/1809.00537">Crowdsourcing Semantic Label Propagation in Relation Classification</a></strong>. <a href="http://fever.ai/">FEVER</a> Workshop at <a href="http://emnlp2018.org/">EMNLP 2018</a>.</li>
<li>Anca Dumitrache, Lora Aroyo and Chris Welty: <strong><a href="https://arxiv.org/abs/1711.05186">False Positive and Cross-relation Signals in Distant Supervision Data</a></strong>. <a href="http://www.akbc.ws/">AKBC</a> Workshop at <a href="http://nips.cc/">NIPS 2017</a>.</li>
<li>Anca Dumitrache, Lora Aroyo and Chris Welty: <strong><a href="http://crowdtruth.org/wp-content/uploads/2017/03/collint17-open-domain.pdf">Disagreement in Crowdsourcing and Active Learning for Better Distant Supervision Quality</a></strong>. <a href="http://collectiveintelligenceconference.org/">Collective Intelligence 2017</a>.</li>
</ul>
<p>Sentence-level data is available in file: <code>|--data/output/aggregated_sentences.csv</code></p>
<p>Worker-level data is available in file: <code>|--data/output/aggregated_workers.csv</code></p>
<p>Raw crowdsourcig data is available in folder: <code>|--data/input/</code></p>
<p>Results of the relation classification model are available in folder: <code>|--data/model_results/</code></p>
<p> </p>
<p>References</p>
<p>(1) Angeli, Gabor, et al. "Combining distant and partial supervision for relation extraction." Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014.</p>
<p>(2) Riedel, Sebastian, et al. "Relation extraction with matrix factorization and universal schemas." Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 2013.</p>
<p>(3) Mintz, Mike, et al. "Distant supervision for relation extraction without labeled data." Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2. Association for Computational Linguistics, 2009.</p>
- …
