Loading...
Loading...
Contributors:
Dataset URI:
10.1371/journal.pone.0325543.t005Vision-language models aim to seamlessly integrate visual and linguistic information for multi-modal tasks, demanding precise semantic alignments between image-text pairs while minimizing the influence of irrelevant data. While existing methods leverage intra-modal and cross-modal knowledge to enhance alignments, they often fall short in sufficiently reducing interference, which ultimately constrains model performance. To address this gap, we propose a novel vision-language model, the threshold-based knowledge integration network (TBKIN), designed to effectively capture intra-modal and cross-modal knowledge while systematically mitigating the impact of extraneous information. TBKIN employs unified scene graph structures and advanced masking strategies to strengthen semantic alignments and introduces a fine-tuning strategy based on threshold selection to eliminate noise. Comprehensive experimental evaluations demonstrate the efficacy of TBKIN, with our best model achieving state-of-the-art accuracy of 73.90% on the VQA 2.0 dataset and 84.60% on the RefCOCO dataset. Attention visualization and detailed result analysis further validate the robustness of TBKIN in tackling vision-language tasks. The model’s ability to reduce interference while enhancing semantic alignments underscores its potential for advancing multi-modal learning. Extensive experiments across four widely-used benchmark datasets confirm its superior performance on two typical vision-language tasks, offering a practical and effective solution for real-world applications.
Keyword(s):
0 dataset; advanced masking strategies; art accuracy; attention visualization; based explicit selection.
MeSH Term(s):
Grants and Funding:
No Grants or Funding Data Available
Associated Publication(s):
Supplementary Data:
No Supplementary Data Available
Related Dataset(s):
No Related Dataset Data Available
The National Library of Medicine (NLM) provides information about biomedical datasets. Inclusion on an NLM Web page does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: NLM Web Policies (nih.gov)