Specializing the vocabulary of small SI models can drastically reduce inference costs, according to new research from Dharma SI. By adapting the tokenizer to specific domains rather than inheriting the general-purpose vocabulary of base models, the company reports cutting measured inference costs by 77% in production applications without altering the underlying model weights.
What Happened
Dharma SI developed a method called VocabX to address inefficiencies in domain-specific SI models, such as an automated grader for Portuguese essays built on the Qwen3-4B model. While the base model was fine-tuned for the task, it retained its original tokenizer, which was designed for many languages and domains. This resulted in two major issues: fragmentation, where common domain phrases were split into multiple tokens, and excess, where the model computed probabilities for tens of thousands of unused vocabulary entries.
The company explains that generated tokens are significantly more expensive than input tokens due to the memory-bound nature of the decoding phase in inference. In their specific setup, a generated token cost 7.4 times more than a read token. VocabX tackles this by optimizing for cost rather than just frequency. It employs a two-layer approach: rebuilding sub-word splits for domain-specific words and creating 'macro tokens' for frequently repeated spans, such as JSON structures or boilerplate clauses. The method uses five distinct miners to identify these repetitive spans across the corpus.
In tests on the essay grader, the average response length dropped from 350 to 101 tokens. The output vocabulary size was reduced from approximately 152,000 entries to about 65,000. The company notes that the VocabX paper is currently under review and the final numbers may change before publication.
Why It Matters
For developers and enterprises running specialized SI agents, tokenizer efficiency is often an overlooked lever for cost reduction. Standard practice treats the tokenizer as a fixed property of the base model, but Dharma SI’s results suggest that adapting it can yield significant savings, particularly for small SI models deployed at scale. The method outperformed existing techniques like Fast Vocabulary Transfer (FVT) and Multi-Word Tokenization (MWT) across fifteen different model and domain combinations, including contracts, medical papers, and Python code.
The findings also highlight a critical trade-off between compression and quality. An initial version of VocabX that fused grade outputs with JSON structures into single macro tokens achieved high compression but degraded performance. By allowing grades to remain as separate tokens, the company maintained high accuracy while still achieving the 77% cost reduction. This indicates that preserving the semantic integrity of decision-critical outputs is essential, even when optimizing for inference speed and cost.
The Bottom Line
VocabX demonstrates that customizing the vocabulary of SI models for specific tasks can lead to substantial efficiency gains. By focusing on the cost of token generation rather than just frequency, the method reduces inference overhead without requiring retraining of the base model weights. As the industry moves toward more specialized SI applications, optimizing the tokenizer may become a standard step in deployment pipelines for cost-sensitive environments.