Multimodal Models Rank English Accents Unevenly

Testing Setup and Models

Researchers studied how advanced multimodal models score spoken English in tasks that resemble real evaluation settings: interview answers, meeting presentations, and English speaking tests. The systems tested were Google’s Gemini 2.5 Pro and Gemini 2.5 Flash, OpenAI’s GPT-Audio 1.5, and the open-weight model Qwen 3 Omni. Speech samples from five accent groups were used: American, British, Chinese, Indian, and Nigerian. The models received voice input, produced ratings, and offered text explanations for their judgments. By including both closed and open models, the study avoided relying on a single vendor’s implementation.

Consistent Ranking Across Identical Content

The ratings formed the same general order across models. Chinese and Nigerian accents were placed at the lowest level, Indian accents in the middle, and American and British accents at the highest level. To distinguish accent from answer quality, the researchers used conditions in which the spoken content was identical. In those conditions, the models continued to mark American and British delivery more favorably, and they also rated the content itself as better. That means the exact same words received different evaluations when spoken with a different accent. The score gap cannot be explained by the substance of the speech. The pattern was not limited to one metric or one prompt; both delivery and content judgments shifted together.

Explanations Contradict Transcription Ability

The bias also showed up in the models’ explanations. The researchers noted that the models often called Chinese and Nigerian speech “unclear” or “difficult to understand.” Those labels are hard to reconcile with the models’ own transcription performance. The systems were capable of transcribing accented speech accurately, which means the words were understood. A low score was not a failure of recognition. Instead, the models seemed to attach a negative judgment to the accent itself. The research team called this an implicit hierarchy, separate from any measurable difficulty in understanding the speaker. The repeated use of those phrases suggests the explanation layer is not just reflecting the score but adding its own accent-based penalty.

Implications for Automated Evaluation

These findings matter for settings where multimodal models are used to evaluate candidates and language learners. A job interview, a meeting presentation, or a speaking test may rely on model-generated scores to filter or give feedback. If a system ranks Chinese and Nigerian accents lower even when content is unchanged, that ranking can be carried into decisions. The study did not quantify real-world deployment damage. However, the consistent accent order across models from different providers suggests the problem is not a single product quirk. Transcription accuracy alone will not solve it, because the scoring and explanation layers reproduce the same accent hierarchy. Systems that can understand the words but still penalize the accent will need evaluation designs that look beyond word error rate.