Can Structured Prompting Overcome Knowledge Bias in Vision-Language Models?

Kyrylo Kolesnichenko; Martas Ambraziunas

Summary

Vision-language models often answer from what they remember instead of looking at the image. Show one an Adidas logo with four stripes and it may still say three. Vo et al. showed this on closed API models with the VLMBias benchmark. I asked two things: does the same bias show up in smaller open-source models, and can more structured prompting fix it?

Setup


Models	Qwen3-VL-8B, InternVL3-8B, Gemma 3-12B, GLM-4.1V-9B-Thinking
Benchmark	VLMBias, 2784 counterintuitive images in 7 domains
Prompting	baseline, debiased, zero-shot chain-of-thought, compositional chain-of-thought, double-check
Statistics	McNemar test with Bonferroni correction

The five prompt templates: baseline, debiased, zero-shot CoT, compositional CoT and double-check — The five prompts. {question} is the original VLMBias counting question.

Results

The open-source models behave like the API ones: 17–22% accuracy and a 65–85% bias-aligned error rate.
Of 16 comparisons against the baseline prompt, 6 stay significant after Bonferroni correction.
The effects depend on the model. Chain-of-thought cut InternVL3’s bias errors from 75% to 65% but made Gemma 3 worse. No prompt helped all four.

Prompting alone is not enough to reliably reduce knowledge bias in VLMs.

Paper

Can Structured Prompting Overcome Knowledge Bias in Vision-Language Models?Kyrylo Kolesnichenko, Martas Ambraziunas.Proceedings of the 31st International Conference on Information Society and University Studies (IVUS 2026), CEUR Workshop Proceedings, 2026 · presented, to appearcode

BibTeX

@inproceedings{kolesnichenko2026structured,
  title     = {{Can Structured Prompting Overcome Knowledge Bias in Vision-Language Models?}},
  author    = {Kolesnichenko, Kyrylo and Ambraziunas, Martas},
  booktitle = {Proceedings of the 31st International Conference on Information Society and University Studies (IVUS 2026)},
  address   = {Kaunas, Lithuania},
  series    = {CEUR Workshop Proceedings},
  year      = {2026},
  note      = {To appear}
}