Benchmarking Concept-Spilling Across Languages in LLMs

The paper investigates "language spilling"—a phenomenon where multilingual LLMs exhibit systematic bias toward dominant language representations when generating content in non-English languages.

The researchers developed a comparative framework measuring how models handle polysemous words across languages. Their methodology ranks model performance based on when they resort to dominant-language meanings: stronger models produce more target-language meanings before failing, while weaker models resort to foreign meanings earlier in generation sequences.

The study evaluated multiple open and closed multilingual LLMs across nine languages using a structured meaning generation task on 100 high-polysemy English words. Results revealed significant variation in semantic robustness across models and languages, providing a ranking system without requiring definitive error attribution. The work contributes both a scalable comparative benchmark for multilingual semantic evaluation and validation methods for developing more linguistically balanced AI systems.