23 research outputs found
Characterizing Bias in Word Embeddings Towards the Development of a Bilingual Analyzer for Detecting Implicit Gender Associations in Philippine Media Texts
No full textGender studies scholars in the Philippines argue that texts from the country’s mass media institutions have been historically complicit in promoting stereotypical gender associations. To mitigate such unfair constructions, natural language processing scholars work on computa- tional models that automate the analysis of gender bias in language. However, not only has such work been absent in the Philippine context, but scholarly efforts have also not yet considered discourses surround- ing non-heterosexual persons. This study built applied word embed- ding association analysis methods in characterizing gendered associa- tions implicitly expressed in documents derived from Philippine mass media. Results show that both corpus-level and word-level biases exist within these documents. At the corpus level, Filipino texts were found to link verbs and action to the male while linking nouns, objects, and social roles to the female. At the semantic level, analyses showed that implicit biases in Philippine mass media construct the heterosexual male as a hedonistic fool, the heterosexual female in terms of her body and physi- cal appearance, the non-heterosexual male as a delusional fool, and the non-heterosexual female in terms of pornographic fetishes. To help min- imize these representations in local media, a tool was also developed to expedite some aspects of the gender and/or sexuality bias review process
Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia
No full textBias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7, 074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets—a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models
Evaluating Gender Bias in Pre-trained Filipino FastText Embeddings
No full textPast studies show that word embeddings can learn gender biases introduced by human agents into the textual corpora used to train these models. However, it has also been shown that some non-English embeddings may actually not capture such biases in their word representations. This study, therefore, aimed to answer the question: Does the publicly available Filipino FastText word embedding contain gender bias? Various iterations of the Word Embedding Association Test and principal component analysis were conducted on the embedding to answer this question. Results show that the Tagalog FastText embedding not only represents gendered semantic information properly but also captures biases about masculinity and femininity collectively held by Filipinos. Specifically, the embedding most strongly associates the female with nouns pertaining to domestic and caregiving roles and the male with verbs relating to strength and their bodies. The study's findings can help determine what next steps need to be undertaken to reduce or eliminate bias from Filipino embeddings
Robust Bias Evaluation with FilBBQ:A Filipino Bias Benchmark for Question-Answering Language Models
Get PDFWith natural language generation becoming a popular use case for language models, the Bias Benchmark for Question-Answering (BBQ) has grown to be an important benchmark format for evaluating stereotypical associations exhibited by generative models. We expand the linguistic scope of BBQ and construct FilBBQ through a four-phase development process consisting of template categorization, culturally aware translation, new template construction, and prompt generation. These processes resulted in a bias test composed of more than 10,000 prompts which assess whether models demonstrate sexist and homophobic prejudices relevant to the Philippine context. We then apply FilBBQ on models trained in Filipino but do so with a robust evaluation protocol that improves upon the reliability and accuracy of previous BBQ implementations. Specifically, we account for models’ response instability by obtaining prompt responses across multiple seeds and averaging the bias scores calculated from these distinctly seeded runs. Our results confirm both the variability of bias scores across different seeds and the presence of sexist and homophobic biases relating to emotion, domesticity, stereotyped queer interests, and polygamy. FilBBQ is available via GitHub
A Novel Interpretability Metric for Explaining Bias in Language Models:Applications on Multilingual Models from Southeast Asia
Get PDFWork on bias in pretrained language models (PLMs) focuses on bias evaluation and mitigation and fails to tackle the question of bias attribution and explainability. We propose a novel metric, the bias attribution score, which draws from information theory to measure token-level contributions to biased behavior in PLMs. We then demonstrate the utility of this metric by applying it on multilingual PLMs, including models from Southeast Asia which have not yet been thoroughly examined in bias evaluation literature. Our results confirm the presence of sexist and homophobic bias in Southeast Asian PLMs. Interpretability and semantic analyses also reveal that PLM bias is strongly induced by words relating to crime, intimate relationships, and helping among other discursive categories—suggesting that these are topics where PLMs strongly reproduce bias from pretraining data and where PLMs should be used with more caution
A Novel Interpretability Metric for Explaining Bias in Language Models:Applications on Multilingual Models from Southeast Asia
Get PDFWork on bias in pretrained language models (PLMs) focuses on bias evaluation and mitigation and fails to tackle the question of bias attribution and explainability. We propose a novel metric, the bias attribution score, which draws from information theory to measure token-level contributions to biased behavior in PLMs. We then demonstrate the utility of this metric by applying it on multilingual PLMs, including models from Southeast Asia which have not yet been thoroughly examined in bias evaluation literature. Our results confirm the presence of sexist and homophobic bias in Southeast Asian PLMs. Interpretability and semantic analyses also reveal that PLM bias is strongly induced by words relating to crime, intimate relationships, and helping among other discursive categories—suggesting that these are topics where PLMs strongly reproduce bias from pretraining data and where PLMs should be used with more caution
A Novel Interpretability Metric for Explaining Bias in Language Models:Applications on Multilingual Models from Southeast Asia
Get PDFWork on bias in pretrained language models (PLMs) focuses on bias evaluation and mitigation and fails to tackle the question of bias attribution and explainability. We propose a novel metric, the bias attribution score, which draws from information theory to measure token-level contributions to biased behavior in PLMs. We then demonstrate the utility of this metric by applying it on multilingual PLMs, including models from Southeast Asia which have not yet been thoroughly examined in bias evaluation literature. Our results confirm the presence of sexist and homophobic bias in Southeast Asian PLMs. Interpretability and semantic analyses also reveal that PLM bias is strongly induced by words relating to crime, intimate relationships, and helping among other discursive categories—suggesting that these are topics where PLMs strongly reproduce bias from pretraining data and where PLMs should be used with more caution
Characterizing Bias in Word Embeddings Towards Analyzing Gender Associations in Philippine Texts
No full textThe steady increase in computational gender bias research has been mostly done on languages for which reliable NLP packages are readily available—such as English, Chinese, and Spanish. This study expands on this area of research by using word embedding bias analysis methods in the Philippine context. To this end, Philippine media textual corpora consisting of 380 million English words and 921 million Filipino words were compiled and used to train FastText embeddings. These embeddings were then subjected to validation and to the Word Embedding Association Test (WEAT) to characterize bias in the embeddings and in the texts they were trained in. Results show that Filipino texts are associated with the heterosexual male by default, but strongest biases relate to the female and the non-heterosexual. Meanwhile, media texts written in English generally have more balanced gender associations compared to texts written in Filipino. Furthermore, the Filipino corpus links action more to the male and objects and social roles to the female. On the other hand, implicitly gendered words in English texts are mostly nouns. These results contribute to demonstrations of how WEAT can be applied in low-resource languages, such as Filipino
Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia
Get PDFBias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7,074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets, a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models
Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia
Get PDFBias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7,074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets, a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models
