6 research outputs found
Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning
No full textThis repository contains resources developed for the paper: Gupta, V., Zhang, S., Vempala, A., He, Y., Choji, T., Srikumar V., Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning. In: Proceeding of the The Association of Computational Linguistic 2022 (ACL ’22), May 2022".
It includes the relevant rows marking for the train set of the InfoTabS dataset (https://infotabs.github.io/) Gupta et. al. 2020 [1].
We followed the protocol of Gupta et al. (2022) [2] which annotated the development and test sets (alpha1, alpha2, alpha3) sets: one table and three distinct hypotheses formed a HIT. We divide the tasks equally into 110 batches, each batch having 51 HITs each having three examples. In total, we collected 81,282 annotations from 90 distinct annotators.
Overall, twenty five annotators completed over 1000 tasks, corresponding to 87.75 % of the examples, indicating a tail distribution with the annotations. Overall, 16,248 training set table-hypothesis pairs were successfully labeled with the evidence rows. On average, we obtain 89.49% F1-score with equal precision and recall for annotation agreement when compared with majority vote. It also includes an annotation template used on the mTurk platform for crowdsourcing. The cited datasets were used in this work. The cited datasets were used in this work.
Files to access the annotation follow the below structure:
annotation_batches
batches_test: contain final results “.csv” files for all the development and test set batches (taken from Gupta et. al. 2022)
batches_train: contain our annotated results “.csv” files for all the train set batches
README.md: contain the readme for the annotation batches details
main_template_row_relevant.html: content the annotation template used for each HIT i.e. marking the relevant row for each instance
annotation_stats.md: Have details of the annotation statistics
release_mturk: contain the release batches details i.e. csv for corresponding batches released
Files to recreate the annotation statistics and pre-processed data:
results_test: contain the pre-processed batch csv for dev and test set each batch. In the dev and test set. The integrated one computes the agreement stats for all the batches.(taken from Gupta et. al. 2022)
results_train: similar to resutls_train expect contain the pre-processed batch csv for train set.
scripts: contain the scripts needed to create the csv in the results_test and results_train sets. The script title denotes the function (the statistic it computes) for the scripts.
src: the scripts use these python files to create the relevant statistics.
References:
[1] InfoTabS: Inference on Tables as Semi-structured Data, Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, Vivek Srikumar, ACL 2020
[2] Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning, Vivek Gupta, Riyaz A. Bhat, Atreya Ghosal, Manish Srivastava, Maneesh Singh, Vivek Srikumar, TACL 2022, presented at ACL 202
Testing for the validity of purchasing power parity theory both in the long-run and the short-run for ASEAN-5
No full text
Testing for purchasing power parity in 21 African countries using several unit root tests
No full textFactors that contribute to academic success for students from low socio-economic backgrounds: a comparative study of two selected schools; one in Saskatoon, Canada, and the other in Barkin-Ladi (GWOL), Nigeria
No full textMy thesis research addresses the factors that contribute to students' academic performance with special reference to children that come from low socio-economic backgrounds. It is a comparative study of two schools: one in Saskatoon, Canada, and the other in Barkin-Ladi, Nigeria. As a child who came from a low socio-economic background, and later as an adult who worked in a school with many students from low socio-economic backgrounds, I wanted to write on this topic. The sampled schools in Saskatoon and Barkin-Ladi were purposively chosen as those that have a considerable number of children from low socio-economic backgrounds. The basic question I tried to answer in my study is how students who come from low socio-economic backgrounds can best be helped to achieve academically.
In my study, I have learned that the insightful and helpful steps on helping students in the sampled school in Saskatoon are the early focus on literacy, responding to data-driven record keeping, the online survey on What Did You Learn In School Today(WDYLIST), the Child Hunger Education Program (CHEP), and the Safety, Teamwork, Attitude, Responsibility, and Respect (STARR) program. In my research findings with the sampled school in Barkin-Ladi, Nigeria, scouting for financial sponsorship, subsidizing school fees, providing educational learning materials, and organizing competitions, debates, and quizzes are essential for helping students from impoverished backgrounds excel in academics.
I discovered in my study that for participants in the sampled school in Saskatoon, Canada, teaching is viewed primarily as a vocation rather than only as a profession. Teacher perception of the profession is important in regards to being dedicated to meeting the needs of students. The study has also showed that there is a strong sense of community and unity of purpose in both sampled schools.
In the sampled school in Barkin-Ladi, Nigeria, the school being a Catholic mission helped makes a big difference in the moral upbringing of the students. As well, the examination promotion policy kept the students alert and working hard so as not to be retained or repeated in the same class. The poverty level in Nigeria cannot be compared to that of Canada. The poverty in Nigeria is so visible that there can be no mistake about who is poor and who is rich even when looking at the schools that the children attend. I have gathered from my study and my life in Nigeria that the government has a good national policy on Education but poor implementation. The sourcing for sponsorship is a big need for children from poor families to be engaged in school. Implementing the Child Hunger and Education Program (CHEP) and Safety, Teamwork, Attitude, Responsibility, and Respect (STARR) programs in the schools in Nigeria will assist students coming from low socio-economic backgrounds.Includes bibliographical references (pages 138-151).
"A thesis submitted to the Faculty of Education for the partial fulfillment of the requirements for the degree of Master of Education.
ULTRA: Unleash LLMs' Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Refinement
No full textStructural extraction of events within discourse is critical since it avails
a deeper understanding of communication patterns and behavior trends. Event
argument extraction (EAE), at the core of event-centric understanding, is the
task of identifying role-specific text spans (i.e., arguments) for a given
event. Document-level EAE (DocEAE) focuses on arguments that are scattered
across an entire document. In this work, we explore the capabilities of open
source Large Language Models (LLMs), i.e., Flan-UL2, for the DocEAE task. To
this end, we propose ULTRA, a hierarchical framework that extracts event
arguments more cost-effectively -- the method needs as few as 50 annotations
and doesn't require hitting costly API endpoints. Further, it alleviates the
positional bias issue intrinsic to LLMs. ULTRA first sequentially reads text
chunks of a document to generate a candidate argument set, upon which ULTRA
learns to drop non-pertinent candidates through self-refinement. We further
introduce LEAFER to address the challenge LLMs face in locating the exact
boundary of an argument span. ULTRA outperforms strong baselines, which include
strong supervised models and ChatGPT, by 9.8% when evaluated by the exact match
(EM) metric
