651 research outputs found
Challenges with Sign Language Datasets for Sign Language Recognition and Translation
Get PDFWork in this paper is part of the SignON project.27 This project has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No. 101017255.Mathieu De Coster's research is funded by the Research Foundation Flanders (FWO Vlaanderen): file number 77410
Leveraging frozen pretrained written language models for neural sign language translation
Get PDFWe consider neural sign language translation: machine translation from signed to written languages using encoder–decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models
Machine Translation from Signed to Spoken Languages: State of the Art and Challenges
Get PDFAutomatic translation from signed to spoken languages is an interdisciplinary
research domain, lying on the intersection of computer vision, machine
translation and linguistics. Nevertheless, research in this domain is performed
mostly by computer scientists in isolation. As the domain is becoming
increasingly popular - the majority of scientific papers on the topic of sign
language translation have been published in the past three years - we provide
an overview of the state of the art as well as some required background in the
different related disciplines. We give a high-level introduction to sign
language linguistics and machine translation to illustrate the requirements of
automatic sign language translation. We present a systematic literature review
to illustrate the state of the art in the domain and then, harking back to the
requirements, lay out several challenges for future research. We find that
significant advances have been made on the shoulders of spoken language machine
translation research. However, current approaches are often not linguistically
motivated or are not adapted to the different input modality of sign languages.
We explore challenges related to the representation of sign language data, the
collection of datasets, the need for interdisciplinary research and
requirements for moving beyond research, towards applications. Based on our
findings, we advocate for interdisciplinary research and to base future
research on linguistic analysis of sign languages. Furthermore, the inclusion
of deaf and hearing end users of sign language translation applications in use
case identification, data collection and evaluation is of the utmost importance
in the creation of useful sign language translation models. We recommend
iterative, human-in-the-loop, design and development of sign language
translation models.Comment: This is the version of the article submitted to peer review to
Universal Access in the Information Society. Please refer to "De Coster, M.,
Shterionov, D., Van Herreweghe, M. et al. Machine translation from signed to
spoken languages: state of the art and challenges. Univ Access Inf Soc
(2023)." for the published and updated versio
Isolated sign recognition from RGB video using pose flow and self-attention
No full textAutomatic sign language recognition lies at the intersection of natural language processing (NLP) and computer vision. The highly successful transformer architectures, based on multi-head attention, originate from the field of NLP. The Video Transformer Network (VTN) is an adaptation of this concept for tasks that require video understanding, e.g., action recognition. However, due to the limited amount of labeled data that is commonly available for training automatic sign (language) recognition, the VTN cannot reach its full potential in this domain. In this work, we reduce the impact of this data limitation by automatically pre-extracting useful information from the sign language videos. In our approach, different types of information are offered to a VTN in a multi-modal setup. It includes per-frame human pose keypoints (extracted by OpenPose) to capture the body movement and hand crops to capture the (evolution of) hand shapes. We evaluate our method on the recently released AUTSL dataset for isolated sign recognition and obtain 92.92% accuracy on the test set using only RGB data. For comparison: the VTN architecture without hand crops and pose flow achieved 82% accuracy. A qualitative inspection of our model hints at further potential of multi-modal multi-head attention in a sign language recognition context
Querying a sign language dictionary with videos using dense vector search
Get PDFTo search for an unknown sign in a sign language dictionary, users typically indicate parameters of the query, e.g., hand shape and signing location. Recent advances in sign language recognition enable video-based sign language dictionary search. In such a system, users can record an unknown sign and retrieve a list of signs that look similar, preferably including the queried sign as one of the top results. We have realized such a system by interpreting it as a dense vector search task. First, we learn a mapping (embedding) from sign videos to a vector space. The dictionary can then be searched by looking for the vectors in this space that are closest to the vector corresponding to the query. We present a proof of concept on a subset of the Flemish Sign Language dictionary. Further research is required to scale up our method to the large vocabularies of entire dictionaries
Pensar las escalas para pensar las luchas: Autor: Mathieu UHEL
No full textA través de un título sugerente, “pensar las escalas para pensar las luchas”, Mathieu Uhel entreteje la construcción teórico-crítica del concepto escala, generada por la geografía radical anglosajona de finales del siglo XX, con la necesidad/utilidad práctica de la escala para concienciar las luchas sociales. El artículo cumple un doble propósito: por un lado, delinear los elementos de lectura sobre el concepto escala; y, con ello, promover la atención de esta problemática en las luchas contemporáneas. En un primer apartado, Uhel ubica las discusiones académicas en torno a la escala, como herramienta metodológica útil para comprender la complejidad de las sociedades capitalistas; en el segundo apartado, el autor avanza la exposición en torno al contexto de la dimensión escalar del imperialismo capitalista; finalmente, el autor se centra en el rol de la actividad política a escala nacional en la tensa relación entre las imposiciones del capital y la lucha social.Por meio de um título sugestivo, “pensando escalas para pensar lutas”, Mathieu Uhel entrelaça a construção teórico-crítica do conceito de escala, gerado pela geografia radical anglo-saxônica do final do século XX, com a necessidade / utilidade prática escala para aumentar a consciência das lutas sociais. O artigo tem um duplo propósito: por um lado, delinear os elementos de leitura sobre o conceito de escala; e, com isso, promover atenção a esse problema nas lutas contemporâneas. Na primeira seção, Uhel localiza as discussões acadêmicas em torno da escala, como uma ferramenta metodológica útil para compreender a complexidade das sociedades capitalistas; na segunda seção, o autor avança a exposição em torno do contexto da dimensão escalar do imperialismo capitalista; por fim, o autor enfoca o papel da atividade política em escala nacional na tensa relação entre as imposições do capital e a luta social.Mathieu Uhel\u27s suggestive title, “Thinking about scales to think about struggles”, he interweaves the theoretical-critical construction of concept scale, generated by radical Anglo-Saxon geography in the late 20th century, with it´s practical utility to social struggles. The article serves two purposes: on the one hand, Uhel locates academic discussion around scale; and, with this, he promotes attention to this problem in contemporary struggles. In the first section, Uhel locates academic discussions around scale, as a useful methodological tool to understand the complexity of capitalist societies; in the second section, the author advances the argument around the context of the scalar dimension of capitalist imperialism; finally, the author focuses on the role of political activity on a national scale in the tense relationship between the impositions of capital and the social movement
- …
