69 research outputs found
Challenges with Sign Language Datasets for Sign Language Recognition and Translation
No full textWork in this paper is part of the SignON project.27 This project has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No. 101017255.Mathieu De Coster's research is funded by the Research Foundation Flanders (FWO Vlaanderen): file number 77410
Leveraging frozen pretrained written language models for neural sign language translation
No full textWe consider neural sign language translation: machine translation from signed to written languages using encoder–decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models
Machine Translation from Signed to Spoken Languages: State of the Art and Challenges
No full textAutomatic translation from signed to spoken languages is an interdisciplinary
research domain, lying on the intersection of computer vision, machine
translation and linguistics. Nevertheless, research in this domain is performed
mostly by computer scientists in isolation. As the domain is becoming
increasingly popular - the majority of scientific papers on the topic of sign
language translation have been published in the past three years - we provide
an overview of the state of the art as well as some required background in the
different related disciplines. We give a high-level introduction to sign
language linguistics and machine translation to illustrate the requirements of
automatic sign language translation. We present a systematic literature review
to illustrate the state of the art in the domain and then, harking back to the
requirements, lay out several challenges for future research. We find that
significant advances have been made on the shoulders of spoken language machine
translation research. However, current approaches are often not linguistically
motivated or are not adapted to the different input modality of sign languages.
We explore challenges related to the representation of sign language data, the
collection of datasets, the need for interdisciplinary research and
requirements for moving beyond research, towards applications. Based on our
findings, we advocate for interdisciplinary research and to base future
research on linguistic analysis of sign languages. Furthermore, the inclusion
of deaf and hearing end users of sign language translation applications in use
case identification, data collection and evaluation is of the utmost importance
in the creation of useful sign language translation models. We recommend
iterative, human-in-the-loop, design and development of sign language
translation models.Comment: This is the version of the article submitted to peer review to
Universal Access in the Information Society. Please refer to "De Coster, M.,
Shterionov, D., Van Herreweghe, M. et al. Machine translation from signed to
spoken languages: state of the art and challenges. Univ Access Inf Soc
(2023)." for the published and updated versio
Isolated sign recognition from RGB video using pose flow and self-attention
No full textAutomatic sign language recognition lies at the intersection of natural language processing (NLP) and computer vision. The highly successful transformer architectures, based on multi-head attention, originate from the field of NLP. The Video Transformer Network (VTN) is an adaptation of this concept for tasks that require video understanding, e.g., action recognition. However, due to the limited amount of labeled data that is commonly available for training automatic sign (language) recognition, the VTN cannot reach its full potential in this domain. In this work, we reduce the impact of this data limitation by automatically pre-extracting useful information from the sign language videos. In our approach, different types of information are offered to a VTN in a multi-modal setup. It includes per-frame human pose keypoints (extracted by OpenPose) to capture the body movement and hand crops to capture the (evolution of) hand shapes. We evaluate our method on the recently released AUTSL dataset for isolated sign recognition and obtain 92.92% accuracy on the test set using only RGB data. For comparison: the VTN architecture without hand crops and pose flow achieved 82% accuracy. A qualitative inspection of our model hints at further potential of multi-modal multi-head attention in a sign language recognition context
Querying a sign language dictionary with videos using dense vector search
No full textTo search for an unknown sign in a sign language dictionary, users typically indicate parameters of the query, e.g., hand shape and signing location. Recent advances in sign language recognition enable video-based sign language dictionary search. In such a system, users can record an unknown sign and retrieve a list of signs that look similar, preferably including the queried sign as one of the top results. We have realized such a system by interpreting it as a dense vector search task. First, we learn a mapping (embedding) from sign videos to a vector space. The dictionary can then be searched by looking for the vectors in this space that are closest to the vector corresponding to the query. We present a proof of concept on a subset of the Flemish Sign Language dictionary. Further research is required to scale up our method to the large vocabularies of entire dictionaries
From scarcity to understanding : transfer learning for the extremely low resource Irish sign language
No full textOne of the most significant challenges to sign language recognition (SLR) today is the low resource nature of sign language datasets, with many datasets being extremely low resource. Transfer learning is therefore a promising, and likely indispensable, method of increasing recognition performance. The use of pose estimation models, which are typically trained on a large and diverse population, can also aid generalization for extremely low resource sign languages. However, research on transfer learning for pose estimation keypoints as inputs has been limited. In this work, we explore transfer learning as a means to improve SLR classification performance for the extremely low resource Irish Sign Language (ISL). We show that transfer learning on larger datasets containing secondary sign languages significantly improves performance on our target sign language, ISL. To understand these results and the attributes that make one dataset better than another for pre-training, we analyse the linguistic relationships between these datasets. We find that certain attributes of datasets are associated with better transfer learning performance. We hope that our findings will not only motivate further research into transfer learning for pose keypoint-based SLR but also act as a practical guide to researchers on choosing the most suitable datasets with which to pre-train models
- …
