Span-Graph Decoding for Grounded Entity Linking in Noisy Image-Text Posts

Abstract

The rapid proliferation of multimedia content on social networking platforms has necessitated the development of advanced natural language processing systems capable of understanding highly unstructured and multimodal data. Grounded entity linking, the task of identifying textual mentions and linking them to a structured knowledge base while grounding them in accompanying visual information, is critical for semantic understanding. However, social media posts are inherently noisy, characterized by grammatical irregularities, abbreviations, missing punctuation, and complex, often ambiguous image-text relationships. Traditional entity linking models, which primarily rely on clean, unimodal text, suffer significant performance degradation when applied to such noisy environments. This paper proposes a comprehensive span-graph decoding methodology designed specifically to tackle the challenges of grounded entity linking in noisy image-text posts. By abandoning traditional sequential token classification in favor of a span-based approach, the proposed system intrinsically handles overlapping mentions and boundaries obscured by noise. Furthermore, by constructing a heterogeneous graph comprising both textual span nodes and visual region nodes, the model facilitates complex cross-modal reasoning through textually described message-passing mechanisms. The evaluation demonstrates that the integration of a flexible span enumerator with a multimodal graph reasoning network significantly improves disambiguation accuracy and boundary detection in high-noise scenarios compared to existing baseline models.