Enterprises that want token-free multilingual models are increasingly turning to byte-level language models to reduce brittleness in noisy or low-resource text. To enter that field – and make it practical at scale – the Allen Institute for AI (AI2) launched BolmoA new family of models that takes advantage of olmo 3 model By “bytefying” them and reusing their backbone and abilities.
According to Ai2, the company launched two versions, Bolmo 7b and Bolmo 1b, which are “the first fully open byte-level language model”. The company said both models performed competitively with – and in some cases outperformed – other byte-level and character-based models.
Byte-level language models operate directly on raw UTF-8 bytes, eliminating the need for predefined vocabularies or tokenizers. This allows them to more reliably handle misspellings, rare languages, and unconventional text – key requirements for moderation, edge deployment, and multilingual applications.
For enterprises deploying AI in multiple languages, noisy user input, or restricted environments, token-free models offer a way to reduce operational complexity. Ai2’s Bolmo is an attempt to make that approach practical on a larger scale – without starting from scratch.
How Bolmo Works and How It Was Created
AI2 said it trained the Bolmo model using its Dolma 3 data mix, which helped it train olmo flagship modeland some open code datasets and character-level data.
The company stated that its goal is to “provide a reproducible, observable blueprint of the robust subword language model in a way that the community can adopt and extend.” To accomplish this goal, Ai2 will release its own checkpoints, code and a complete paper To help other organizations build byte-level models on top of their Olmo ecosystem.
Since training a byte-level model entirely from scratch could be costly, the AI2 researchers instead chose to perform bytefi on the existing Olmo 3 7B checkpoint in two stages.
In the first stage, Ai2 froze OLMO3 transformers so that they only train certain parts, such as the local encoder and decoder, the boundary predictor, and the language modeling core. It was designed to be “cheap and fast” and requires only 9.8 billion tokens.
The next step unfreezes the model and trains it with additional tokens. AI2 said the byte-level approach allows Bolmo to avoid the lexical constraints that limit traditional subword models.
Strong performance among your peers
Byte-level language models are not as mainstream as small language models or LLMs, but it is a growing area in research. Meta releases its BLT architecture The goal of last year’s research was to introduce a model that is robust, processes raw data, and does not rely on fixed terminology.
Other research models in this area Include ByT5, Stanford’s MrT5And dog’s,
AI2 evaluated Bolmo using its assessment suite covering math, STEM reasoning, question answering, general knowledge, and code.
Bolmo 7B showed strong performance, outperforming character-centric benchmarks such as CUTE and EXECUTE, and also improved accuracy compared to the base LLM Olmo 3.
The Bolmo 7B outperformed comparably sized models in coding, mathematics, multiple choice QA, and character-level understanding.
Why might enterprises choose the byte-level model?
Enterprises find value in a hybrid model structure by using a mix of models and model sizes.
Ai2 makes the case that organizations should also consider the byte-level model not only for robustness and multilingual understanding, but because it “naturally plugs into the existing model ecosystem.”
“A key benefit of the dynamic hierarchical setup is that compression becomes a toggleable knob,” the company said.
For enterprises already running heterogeneous model stacks, Bolmo suggests that byte-level models may no longer be purely academic. By retrofitting a robust subword model rather than training from scratch, AI2 is signaling a lower-risk path forward for organizations that want robustness without abandoning existing infrastructure.
<a href