#'mmBERT

'mmBERT is an encoder pretrained on 3 trillion tokens across 1,833 languages that runs 2–4× faster than previous multilingual encoders and supports 8k token contexts. It combines annealed language learning, inverse masking, and model merging to boost performance on high-resource and low-resource benchmarks.'