GameSmith: Scaling Zero-Sum Self-Play with Synthesised Games

What happened

The authors developed GameSmith, an approach that synthesizes two-player zero-sum games using templates, verifies them through multiple methods, and uses these games to train LLMs via self-play reinforcement learning with the CRAB advantage estimator. The method demonstrated improved performance on reasoning benchmarks compared to existing approaches.

Why it matters

This research advances AI training methodologies by creating scalable, verifiable game environments for self-play, which can enhance general reasoning capabilities in language models. The approach addresses limitations in existing game-based training pipelines.

• Synthesized game environments can scale zero-sum self-play training
• Verification processes improve reliability of generated games
• CRAB advantage estimator enhances reinforcement learning performance
• Model distillation improves efficiency without significant performance loss
• Increasing game diversity improves transfer learning effectiveness

The bigger picture

The method enables more efficient and scalable training of LLMs through game-based self-play, potentially improving reasoning capabilities across various AI applications. The verification process increases reliability of training environments.

GameSmith presents a scalable framework for generating and verifying zero-sum games to train LLMs through self-play, demonstrating improved reasoning capabilities through both distillation and reinforcement learning techniques.

Technical details

• Game synthesis using templates with fixed code components
• Seed-paired replay verification method
• CRAB advantage estimator with shrinkage-based return estimation
• Model distillation from Qwen3-32B to Qwen3-4B-Base
• Multiple game variant generation (encoding, rule, size)

FAQ