Evaluating Semantic Relation Knowledge through Bidirectional Relation Prediction from Gloss Pairs
Semantic relations organise word senses and are fundamental to lexical semantics. In order to evaluate semantic relation knowledge of large language models (LLMs), we propose a challenging task in which an LLM predicts the relation between two glosses (definitions of senses) presented in two…
This study introduces a task to evaluate large language models' (LLMs) understanding of semantic relations by predicting relationships between word sense definitions (glosses) presented in different orders.
What happened
Researchers developed a task where LLMs predict semantic relations between glosses presented in two orders, testing their ability to recognize symmetric/asymmetric relationships. The study evaluated 17 LLMs across multiple languages, finding significant limitations in their semantic relation understanding.
Why it matters
The findings highlight critical gaps in LLMs' lexical semantic knowledge, particularly for asymmetric relations, which impacts applications requiring precise semantic understanding. The cross-lingual evaluation also reveals challenges in transferring semantic knowledge across languages.
- LLMs exhibit significant limitations in understanding asymmetric semantic relations
- Cross-lingual semantic knowledge transfer remains challenging for current models
- The bidirectional evaluation task offers a new method for assessing semantic understanding
The bigger picture
The study underscores the need for improved semantic relation modeling in LLMs and provides a benchmark for evaluating cross-lingual semantic knowledge, influencing future research on language model capabilities.
The research demonstrates that current LLMs struggle with fundamental semantic relation tasks, particularly for asymmetric relationships, and emphasizes the need for better evaluation frameworks to improve semantic understanding in AI systems.
Technical details
- Bidirectional relation prediction task
- Multilingual gloss dataset (5 languages)
- Macro-averaged F1 score evaluation
- Zero-shot and five-shot learning settings
- Hypernymy (is-a) and holonymy (part-of) relations
FAQ
What is the main contribution of this study?
The study introduces a bidirectional relation prediction task to evaluate LLMs' semantic relation knowledge and provides a multilingual dataset for this purpose.
What performance metrics were used?
The study used macro-averaged F1 scores to evaluate model performance in zero-shot and five-shot settings against random and neural baselines.
Which semantic relations were most challenging for LLMs?
LLMs had particular difficulty with hypernymy (is-a) and holonymy (part-of) relations, especially in maintaining bidirectional consistency.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv - cs.CL (NLP) on Oct 14, 2026.
Comments
Sign in to join the discussion.