Finetuned Multimodal Language Models Are
High-Quality Image-Text Data Filters


Weizhi Wang1, Khalil Mrini2, Linjie Yang2, Sateesh Kumar2, Yu Tian2, Xifeng Yan1, Heng Wang2

1 UCSB   2 Bytedance US
arXiv Code Dataset Model

Abstract

We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g. CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e. CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter.

Illustration of Fine-Tuned MLM as Image-Text Quality Scorer

Figure 1. CLIPScore versus Image-Text Matching score generated by MLM Filter.

Fine-Tuning MLM as Image-Text Data Filter

Figure 2: Illustration of the pipeline of fine-tuning MLM Filter and employing it for data filtering

We propose to adopt fine-tuned Multimodal Language Model as effective data filters to select high-quality image-text data to promote the VLM pre-training, which involves three stages:

  • Construct multimodal instruction tuning data on proposed quality scoring tasks to fine-tune MLM to realize accurate quality assessment;
  • Adopt the fine-tuned MLM Filter to generate quality scores for each data point in the data pool and then select the high-quality data;
  • Pre-train VLMs using the filtered dataset and evaluate the pre-trained VLMs on downstream tasks to demonstrate the effectiveness of the proposed filtering method.

Constructing Multimodal Instructons for Quality Scoring Tasks

Defining Metrics for Image-Text Quality Assessment. We defined four different data quality metrics for comprehensive quality assessment of image-text paired data:

  • Image-Text Matching (ITM): the ITM metric focuses on evaluating whether the image caption accurately represents the main features and objects of the image and captures its primary theme. The fine-tuned MLM data filter can explicitly generate the ITM score on a scale of 100.
  • Object Detail Fulfillment (ODF): the ODF metric focuses on evaluating whether the image caption provides detailed descriptions of objects that align with the image. Specifically, ODF assesses if the caption sufficiently describes the properties of the objects in the image, e.g. number, color, size, position, shape, etc. Compared with the ITM metric, the ODF metric focuses more on the fine-grained alignment between the detailed object properties in the image and the ones described in the corresponding caption.
  • Caption Text Quality (CTQ): the CTQ metric focuses on evaluating the text quality of image caption based on the grammatical correctness, diversity of vocabulary (e.g. the range and uniqueness of words), fluency (\eg smoothness and natural flow of sentences), readability, length, and structure.
  • Semantic Understanding (SU): the SU metric focuses on determining if the image caption provides additional semantic information that is not readily apparent just from the image itself. Such auxiliary semantic information can be (1) the professions of persons in the image; (2) the locations, addresses, festivals, country names, city names; (3) the names or entities of buildings, people, bird species, animal breeds, car models, engines in the image; (4) the social relationships between the people in the image, i.e. lovers, parent & child. We suggest that adopting SU metric for data filtering can select image-text pairs with auxiliary semantics, which can further enhance the commonsense reasoning capability of pre-trained VLMs.

Other Data Construction Designs. We use CC12M image-text data as the data resources and then sample 10k image-text pairs based on the clustered text embeddings of captions. We prompt the GPT-4/GPT-4V models using the following templates to construct initial 40k instruction data on 4 quality assessment tasks. Then we uniformly sample 4k instructions from initial 40k intructions based on the integer scores to avoid scoring bias brought by imbalanced fine-tuning. Additionally, we sample another 46k instruction data from LLaVA-665k multimodal instruction set to ensure the diversity of final 50k instruction dataset. 


{image}
Text Caption: {caption}

Please evaluate if the provided text caption accurately represents the main features and objects of the image. 
The caption doesn't need to detail every aspect of the image, but it should capture its primary theme. 
Rate the overall quality of the text caption's match to the image on a scale of 1-100, considering the criteria mentioned.
A higher score indicates higher level of image text matching. 
Ensure that your scoring is nuanced and uses the entire range from 0 to 100, reflecting the subtle differences. 
The score should be given as an integer, with each number between 0 and 100 considered as a potential score, avoiding the tendency to round to multiples of 10.  
Please first output a single line containing the value indicating the scores. 
In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias.

Instruction-Tune Multimodal Language Model as Data Filter. We select LLaVA-13b as the foundation MLM for instruction-tuning on sampled 50k instructions. We directly take the pre-trained checkpoint after pre-training stage-1 of LLaVA and only change the instruction data in stage-2. Please check out our MLM-Filter model at [Model Zoo].

Results

DataComp Medium

Figure 3. Zero-shot performance of CLIP models pre-trained using baseline filtering methods and proposed MLM-FILTER on Medium scale pools of the DataComp benchmark. AND represents the combination of ITM and ODF metrics using AND operation.

Figure 4. Zero-shot performance of CLIP models pre-trained using baseline filtering methods and proposed MLM-FILTER on Large scale pools of the DataComp benchmark.

Figure 5. Pearson and Spearman correlations between human-labeled quality scores and scores generated by MLM-Filter and CLIP. Images are scored on a scale of 100 for our MLMFilter, while CLIPScore is also normalized to the scale of 100. The ∗ denotes significant correlations at p < 0.05.

The evaluation results demonstrate the prposed MLM-FILTER can work as more effective filtering method than CLIPScore filter. Additionally, we can draw the following auxiliary conclusions from the results:

  • MLM Filter learned from GPT-4V performs better on ImageNet related datasets than the MLM Filter learned from GPT-4.
  • The optimal filtering metric varies for fine-tuned MLM Filter learned from different teacher models.
  • Image-Text Matching is the best filtering metric for retrieval tasks. The proposed MLM Filter also achieves SOTA performance on retrieval tasks compared with other filtering baselines on DataComp Leaderboard.
  • Combing different quality metrics effectively filters and identifies image-text pairs of better quality. Combing ITM and ODF metrics for filtering yields the best average performance on DataComp Medium.
  • Our proposed MLM-FILTER scores are significantly aligned and correlated with human quality scores, while CLIPScore does not demonstrate such correlations.

BibTeX


  @article{mlm-filter,
    title={Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters}, 
    author={Wang, Weizhi and Mrini, Khalil and Yang, Linjie and Kumar, Sateesh and Tian, Yu and Yan, Xifeng and Wang, Heng},
    publisher={arXiv preprint arXiv:2403.02677},
    year={2024},
  }

Acknowledgement

This website is adapted from Nerfies, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. The foundation multimodal language models of our proposed MLM Filter are based on the OpenAI CLIP, Meta LLaMA-2 and its fine-tuned model Vicuna. We also thank LLaVA team to propose the great multimodal language model architecture as well as providing the codebase for further development.

Usage and License Notices: The data, code and checkpoint is intended and licensed for research use only. They are also restricted to uses that follow the license agreement of CLIP, LLaMA, Vicuna and GPT-4. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.