Let’s Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models

arXiv:2601.20419v1 Announce Type: cross
Abstract: Recent research has shown that aligning fine-grained text descriptions with localized image patches can significantly improve the zero-shot performance of pre-trained vision-language models (e.g., CLIP). However, we find that both fine-grained text descriptions and localized image patches often contain redundant information, making text-visual alignment less effective. In this paper, we tackle this issue from two perspectives: emphView Refinement and emphDescription refinement, termed as textittextbfBi-refinement for textbfFine-grained textbfText-visual textbfAlignment (BiFTA). emphView refinement removes redundant image patches with high emphIntersection over Union (IoU) ratios, resulting in more distinctive visual samples. emphDescription refinement removes redundant text descriptions with high pairwise cosine similarity, ensuring greater diversity in the remaining descriptions. BiFTA achieves superior zero-shot performance on 6 benchmark datasets for both ViT-based and ResNet-based CLIP, justifying the necessity to remove redundant information in visual-text alignment.