Weakly-supervised Fingerspelling Recognition in British Sign Language Videos - ANR - Agence nationale de la recherche Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Weakly-supervised Fingerspelling Recognition in British Sign Language Videos

Résumé

The goal of this work is to detect and recognize sequences of letters signed using fingerspelling in British Sign Language (BSL). Previous fingerspelling recognition methods have not focused on BSL, which has a very different signing alphabet (e.g., two-handed instead of one-handed) to American Sign Language (ASL). They also use manual annotations for training. In contrast to previous methods, our method only uses weak annotations from subtitles for training. We localize potential instances of fingerspelling using a simple feature similarity method, then automatically annotate these instances by querying subtitle words and searching for corresponding mouthing cues from the signer. We propose a Transformer architecture adapted to this task, with a multiple-hypothesis CTC loss function to learn from alternative annotation possibilities. We employ a multi-stage training approach, where we make use of an initial version of our trained model to extend and enhance our training data before re-training again to achieve better performance. Through extensive evaluations, we verify our method for automatic annotation and our model architecture. Moreover, we provide a human expert annotated test set of 5K video clips for evaluating BSL fingerspelling recognition methods to support sign language research.

Dates et versions

hal-03981734 , version 1 (10-02-2023)

Identifiants

Citer

K R Prajwal, Hannah Bull, Liliane Momeni, Samuel Albanie, Gül Varol, et al.. Weakly-supervised Fingerspelling Recognition in British Sign Language Videos. British Machine Vision Conference (BMVC) 2022, Nov 2022, London, United Kingdom. ⟨hal-03981734⟩
59 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More