A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks

TY - GEN

T1 - A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks

AU - Yu, Haorui

AU - Ruiz-Dolz, Ramon

AU - Yi, Qiufeng

N1 - ©2025 Association for Computational Linguistics

PY - 2025/11

Y1 - 2025/11

N2 - This study aims to test and evaluate the capabilities and characteristics of current mainstream Visual Language Models (VLMs) in generating critiques for traditional Chinese painting. To achieve this, we first developed a quantitative framework for Chinese painting critique. This framework was constructed by extracting multi-dimensional evaluative features covering evaluative stance, feature focus, and commentary quality from human expert critiques using a zero-shot classification model. Based on these features, several representative critic personas were defined and quantified. This framework was then employed to evaluate selected VLMs such as Llama, Qwen, or Gemini. The experimental design involved persona-guided prompting to assess the VLM’s ability to generate critiques from diverse perspectives. Our findings reveal the current performance levels, strengths, and areas for improvement of VLMs in the domain of art critique, offering insights into their potential and limitations in complex semantic understanding and content generation tasks.

AB - This study aims to test and evaluate the capabilities and characteristics of current mainstream Visual Language Models (VLMs) in generating critiques for traditional Chinese painting. To achieve this, we first developed a quantitative framework for Chinese painting critique. This framework was constructed by extracting multi-dimensional evaluative features covering evaluative stance, feature focus, and commentary quality from human expert critiques using a zero-shot classification model. Based on these features, several representative critic personas were defined and quantified. This framework was then employed to evaluate selected VLMs such as Llama, Qwen, or Gemini. The experimental design involved persona-guided prompting to assess the VLM’s ability to generate critiques from diverse perspectives. Our findings reveal the current performance levels, strengths, and areas for improvement of VLMs in the domain of art critique, offering insights into their potential and limitations in complex semantic understanding and content generation tasks.

UR - https://doi.org/ 10.18653/v1/2025.findings-emnlp

U2 - 10.18653/v1/2025.findings-emnlp.103

DO - 10.18653/v1/2025.findings-emnlp.103

M3 - Conference contribution

SN - 9798891763357

SP - 1945

EP - 1971

BT - Findings of the Association for Computational Linguistics

A2 - Christodoulopoulos, Christos

A2 - Chakraborty, Tanmoy

A2 - Rose, Carolyn

A2 - Peng, Violet

PB - Association for Computational Linguistics

CY - China

T2 - The 2025 Conference on Empirical Methods in Natural Language Processing

Y2 - 4 November 2025 through 9 November 2025

ER -