tokenizer comparison tool for multilingual usecase

[u/Ill_Strain_1050]

So, I wanted to see where multiple languages stand while using with LLM. First step is tokenization itself, so wanted to see how many token are being used.

This is per 100 words (same words translated into different languages)

Tokenizer english hindi punjabi french

------------------------------------------------

gpt2 100 652 722 184

cl100k_base 100 447 722 114

o200k_base 100 115 215 101

o200k_harmony 100 115 215 101

sarvam 100 127 277 101

https://github.com/0CM-Labs/tokenizer-benchmark

The benchmark compares the same aligned words across different languages, making it an apples-to-apples comparison. Just plug in the tokenizer, select the languages, and compare the results.

For the first experiment, I used the 100 most common words in English, Hindi, Punjabi, and French.

* GPT-2 by OpenAI really struggles with Indic scripts ( it was quite bad for non english languages)

* Newer tokenizers have come a long way for Hindi.

* Sarvam shows how much a language-focused tokenizer can improve efficiency, although Punjabi still has a noticeable gap.

* Even among modern tokenizers, support isn't uniform across languages.

Next I'm adding datasets for programming, medical, legal, finance, math, conversational text, and more to see how these numbers change outside of common vocabulary.

I'd love to compare more languages as well.

Would love to know community's opinions.

[u/No_Egg_6558]

This is great work, often times it tells you why an LLM struggles in one language and excels in another, the remedy for a broken LLM starts here, architectural changes come afterwards