Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters
IceBERT, the reference encoder for Icelandic, found that a cleaned web crawl is enough to train a strong model and that starting from a multilingual model can help on some tasks. We revisit both findings with the ModernBERT architecture.
arXiv - cs.CL (NLP)2 min read