Adapting Vision-Language Models for E-commerce Understanding at Scale |

Matteo Nulli, Vladimir Orshulevich, Tala Bazazo, Christian Herold, Michael Kozielski, Marcin Mazur, Szymon Tuzel, Cees G. M. Snoek, Seyyed Hadi Hashemi, Omar Javed, Yannick Versley, Shahram Khadivi

eBay, University of Amsterdam

📄 Paper | 📝 Blogpost

Accepted to EACL 2026 Industry Track

Motivation

If you’ve ever shopped online for something even mildly specific—say, a “navy waterproof trail running jacket with taped seams”—you already know the problem: product understanding isn’t just text. It’s photos. It’s packaging. It’s messy seller-generated descriptions. It’s half-structured attributes that sometimes help and often don’t. And it’s usually multiple images that each reveal different parts of the truth.

E-commerce product understanding is, by nature, deeply multimodal.

Yet most production shopping systems still lean heavily on text matching—titles, short descriptions, maybe a few extracted specs—because it’s reliable, scalable, and easy to index. The issue is that customers don’t shop that way. People compare photos for color, texture, size cues, labels, logos, and packaging details, and then reconcile what they saw with whatever the listing claims.

Meanwhile, the model landscape has changed fast. Large Language Models have started showing strong performance on e-commerce tasks, especially when you tailor them to the domain—either through adaptation strategies or domain-aware training recipes (1, 2, 3, 4). But translating that success into the vision–language setting is not automatic—and that gap is exactly what this work is about.

Figure 1: Output of our E-commerce Adapted VLMs compared against same size LLaVA-OneVision. We show our models ability to more faithfully extract attributes from e-commerce items. In red, we highlight wrong model predictions that are neither tied to the image nor valid item attributes.

General-purpose Vision–Language Models (VLMs) have reached a point where multimodal assistants are finally practical. Modern models can jointly reason over images and text, follow instructions, and generalize across tasks, building on strong vision and language backbones (5, 6). Recent families—such as LLaVA and its successors (7, 9, 10), Qwen3-VL (11), InternVL4 (12), and Gemma3 (13)—demonstrate how quickly the field is advancing. However, e-commerce remains a difficult target. Shopping scenarios demand attribute-centric reasoning, aggregation across multiple images, robustness to noisy seller content, and strict instruction following—requirements that go well beyond standard captioning or VQA. While multimodal benchmarks exist (15, 16, 17), they only partially capture these shopping-specific challenges.

We study whether strong e-commerce performance truly requires a custom language model, or whether targeted adaptation of vision components is sufficient. Through extensive ablations across different visual and text decoders, combined with efficient adaptation techniques such as low-rank updates (22), we show that careful multimodal adaptation can yield substantial gains. In parallel, we introduce a new evaluation suite designed around real online retail use-cases. It emphasizes deep product understanding, strict instruction following, and multi-image reasoning—dimensions that are underrepresented in existing benchmarks.

Our main finding is simple: general-purpose VLMs can be effectively adapted to e-commerce with targeted, reproducible strategies that preserve broad multimodal competence. This enables meaningful improvements on shopping tasks without sacrificing performance in other domains, and provides a clearer path toward realistic multimodal shopping assistants.

Methodology

Figure 3:

Results

Figure 3:

Figure 4:

Figure 5:

Citation

If you use this work, please cite:

@inproceedings{nulli2025objectguided,
title={Object-Guided Visual Tokens: Eliciting Compositional Reasoning in Multimodal Language Models},
author={Matteo Nulli and Ivona Najdenkoska and Mohammad Mahdi Derakhshani and Yuki M Asano},
booktitle={EurIPS 2025 Workshop on Principles of Generative Modeling (PriGM)},
year={2025},
url={https://openreview.net/forum?id=yvY1T3hHEQ}
}

References

Huang Irene, Lin Wei, Mirza M. Jehanzeb, Hansen Jacob A., Doveh Sivan, Butoi Victor Ion, Herzig Roei, Arbelle Assaf, Kuehne Hilde, Darrell Trevor, Gan Chuang, Oliva Aude, Feris Rogerio, Karlinsky Leonid. (2024). Conme: Rethinking Evaluation of Compositional Reasoning for Modern VLMs. arXiv preprint arXiv:2406.08164.

Tong Shengbang, Liu Zhuang, Zhai Yuexiang, Ma Yi, LeCun Yann, Xie Saining. (2024). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. arXiv preprint arXiv:2401.06209.

Liu Haotian, Li Chunyuan, Wu Qingyang, Lee Yong Jae. (2023). Visual Instruction Tuning. arXiv preprint arXiv:2304.08485.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. Preprint, arXiv:2408.03326.

Bai Shuai, Chen Keqin, Liu Xuejing, Wang Jialin, Ge Wenbin, Song Sibo, Dang Kai, Wang Peng, Wang Shijie, Tang Jun, Zhong Humen, Zhu Yuanzhi, Yang Mingkun, Li Zhaohai, Wan Jianqiang, Wang Pengfei, Ding Wei, Fu Zheren, Xu Yiheng, Ye Jiabo, Zhang Xi, Xie Tianbao, Cheng Zesen, Zhang Hang, Yang Zhibo, Xu Haiyang, Lin Junyang. (2025). Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923.

OpenGVLab-Team. (2024). InternVL2: Better Than the Best—Expanding Performance Boundaries of Open-Source Multimodal Models with the Progressive Scaling Strategy. Blog post. URL https://internvl.github.io/blog/2024-07-02-InternVL-2.0/.

Gemma-Team. (2025). Gemma 3 Technical Report. arXiv preprint arXiv:2503.19786.

Yuksekgonul Mert, Bianchi Federico, Kalluri Pratyusha, Jurafsky Dan, Zou James. (2023). When and Why Vision-Language Models Behave Like Bags-of-Words, and What to Do About It? arXiv preprint arXiv:2210.01936.

Nulli Matteo, Ibrahimi Anesa, Pal Avik, Lee Hoshe, Najdenkoska Ivona. (2024). In-Context Learning Improves Compositional Understanding of Vision-Language Models. In ICML 2024 Workshop on Foundation Models in the Wild. arXiv preprint arXiv:2407.15487.

Awal Rabiul, Ahmadi Saba, Zhang Le, Agrawal Aishwarya. (2025). Vismin: Visual Minimal-Change Understanding. arXiv preprint arXiv:2407.16772.

Tong Shengbang, Brown Ellis, Wu Penghao, Woo Sanghyun, Middepogu Manoj, Akula Sai Charitha, Yang Jihan, Yang Shusheng, Iyer Adithya, Pan Xichen, Wang Austin, Fergus Rob, LeCun Yann, Xie Saining. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. arXiv preprint arXiv:2406.16860.