HELM Arabic Enterprise
We present HELM Arabic Enterprise, a leaderboard for transparent, reproducible evaluation of large language models on Arabic-language benchmarks designed around enterprise use cases.
Alpaca Blog7 min read
Alpaca
10 stories
HELM Arabic
As part of our efforts to better understand the multilingual capabilities of large language models (LLMs), we present HELM Arabic, a leaderboard for transparent and reproducible evaluation of LLMs on Arabic language benchmarks. This leaderboard was produced in collaboration with Arabic.AI.
Alpaca Blog5 min readHELM Long Context
We introduce the HELM Long Context leaderboard for transparent, comparable and reproducible evaluations of long context capabilities of recent models.
Alpaca Blog18 min read
Reliable and Efficient Amortized Model-Based Evaluation
TLDR: We enhance the reliability and efficiency of language model evaluation by introducing IRT-based adaptive testing, which has been integrated into the HELM framework.
Alpaca Blog5 min readSurprisingly Fast AI-Generated Kernels We Didn’t Mean to Publish (Yet)
We have some very fast AI-generated kernels in pure CUDA-C without using libraries and DSLs such as CUTLASS and Triton. They are performing close to or in some cases even beating the standard expert-optimized production kernels shipped in PyTorch. Some of our highlighted results:
Alpaca Blog28 min read
BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems
We introduce BountyBench, a benchmark featuring 25 systems with complex, real-world codebases, and 40 bug bounties that cover 9 of the OWASP Top 10 Risks.
Alpaca Blog9 min readHELM Capabilities: Evaluating LMs Capability by Capability
Introducing HELM Capabilities, a benchmark that evaluates language models across a curated set of key capabilities, providing a comparison of their strengths and weaknesses.
Alpaca Blog34 min read
General-Purpose AI Needs Coordinated Flaw Reporting
Today, we are calling for AI developers to invest in the needs of third-party, independent researchers, who investigate flaws in AI systems. Our new paper advocates for a new standard of researcher protections, reporting and coordination infrastructure.
Alpaca Blog6 min readHELM Safety: Towards Standardized Safety Evaluations of Language Models
We introduce HELM Safety v1.0 as a collection of 5 safety benchmarks spanning 6 risk categories (violence, fraud, discrimination, sexual content, harassment, deception) and evaluate 24 prominent language models as an ongoing effort to standardize safety evaluations.
Alpaca Blog14 min read
Advancing Customizable Benchmarking in HELM via Unitxt Integration
The Holistic Evaluation of Language Models (HELM) framework is an open source framework for reproducible and transparent benchmarking of language models that is widely adopted by academia and industry.
Alpaca Blog4 min read
Showing 10 of 10 stories
You're all caught up.
AiByts Weekly Brief
Get the top AI news and tools straight to your inbox every week.
Weekly · AI news
Join 15,000+ AI enthusiasts