Neither True nor False: Do LLMs Represent Truth-Value Gaps in Plural Predication?

What happened

Researchers tested seven open-weight LLMs on the HomogeneityNLI dataset to evaluate their ability to handle truth-value gaps in plural predication. The study found that models consistently collapse trivalent logic into bivalence, misclassifying cases where only some group members satisfy a predicate.

Why it matters

Understanding truth-value gaps is critical for improving LLMs' semantic accuracy in real-world applications involving collective predicates, such as legal documents, scientific reporting, and conversational agents.

• LLMs struggle to represent trivalent logic in plural predication tasks
• The Bivalence Bias persists across different prompting strategies
• Truth-value gaps require specialized training for accurate modeling
• Current models prioritize bivalent logic over semantic accuracy in plural contexts
• NLP systems need better handling of collective predicates for real-world applications

The bigger picture

The findings suggest current LLMs may misinterpret collective predicates in applications requiring precise semantic analysis, highlighting the need for improved logical reasoning capabilities in language models.

The study demonstrates that while LLMs have some understanding of plural predication, they fail to represent the trivalent logic required for accurate truth-value evaluation in cases involving the Homogeneity Gap.

Technical details

• Homogeneity Gap: Semantic phenomenon where plural definites and their negations can be neither true nor false
• Bivalence Bias: Tendency of models to assign strict true/false values to sentences that should have intermediate truth values
• HomogeneityNLI: Diagnostic dataset with minimal-pair items testing logical conditions in plural predication
• Zero-shot setting: Evaluation without task-specific training data
• Chain-of-thought prompts: Instructional techniques to encourage step-by-step reasoning

FAQ