Entity-aware cross-modal pretraining for Knowledge-Based Visual Question Answering
Résumé
Knowledge-Aware Visual Question Answering about Entities (KVQAE) is a recent multimodal retrieval task aiming to answer visual questions about named entities from a multimodal knowledge base. In this context, we focus more particularly on cross-modal retrieval and propose to inject information about entities in the representations of both texts and images during their building through two pretraining auxiliary tasks, namely entity-level masked language modeling and entity type prediction objectives. We show competitive performance over existing approaches on 3 KVQAE standard benchmarks, revealing the interest of raising entity awareness during cross-modal pretraining and specifically for the KVQAE task 3
Origine | Fichiers produits par l'(les) auteur(s) |
---|