Robust Image-Text Matching Through Capacity-Aligned Dense-to-Sparse Feature Distillation

Abstract

The rapid advancement of vision-language modeling has significantly improved the performance of multi-modal systems, particularly in the domain of image-text matching. However, achieving state-of-the-art accuracy typically requires computationally expensive architectures that process dense feature representations across multiple cross-attention layers. This complexity poses significant challenges for deployment in resource-constrained environments. While sparse representation learning offers a promising avenue for reducing computational overhead, directly distilling knowledge from a high-capacity dense teacher model to a low-capacity sparse student model often results in severe performance degradation. This degradation is primarily attributed to the representational capacity gap and the inherent topological differences between dense and sparse feature spaces. To address these challenges, this paper proposes a novel framework titled Capacity-Aligned Dense-to-Sparse Feature Distillation. The proposed methodology introduces a structural alignment mechanism that explicitly bridges the dimensionality and distribution gaps between teacher and student representations prior to the distillation process. By projecting dense contextualized features into an intermediate capacity-aligned subspace, the framework enables the sparse student model to efficiently absorb rich cross-modal relationships without being overwhelmed by representational complexity. Extensive experimental evaluations conducted on standard benchmark datasets demonstrate that the proposed method significantly outperforms traditional distillation techniques. The framework recovers the majority of the performance drop typically associated with feature sparsification, establishing a new Pareto frontier for the trade-off between inference efficiency and retrieval accuracy in multi-modal architectures.