Scaling Up Sign Spotting Through Sign Language Dictionaries - ANR - Agence nationale de la recherche Accéder directement au contenu
Article Dans Une Revue International Journal of Computer Vision Année : 2022

Scaling Up Sign Spotting Through Sign Language Dictionaries

Résumé

Abstract The focus of this work is sign spotting –given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) watching existing footage which is sparsely labelled using mouthing cues; (2) reading associated subtitles (readily available translations of the signed content) which provide additional weak-supervision ; (3) looking up words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BslDict , to facilitate study of this task. The dataset, models and code are available at our project page.

Dates et versions

hal-03981735 , version 1 (10-02-2023)

Identifiants

Citer

Gül Varol, Liliane Momeni, Samuel Albanie, Triantafyllos Afouras, Andrew Zisserman. Scaling Up Sign Spotting Through Sign Language Dictionaries. International Journal of Computer Vision, 2022, 130 (6), pp.1416-1439. ⟨10.1007/s11263-022-01589-6⟩. ⟨hal-03981735⟩
15 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More