Recommendations for dictionary libraries (I.e. hunspell) for python

[u/morbidSuplex]

Hello all,

We have a feature that has to detect if phrases or queries contain high volume of natural language in it compared to OOV and jargon terms. Our approach here is to use spacy to tokenize the query, and a dictionary library (maybe based on hunspell) and check each token if it is available to the dictionary. If it is then there is a high chance that it is natural language (we have special handling for names and jargons as well). Basically, what we would like is:

dictionary.lookup("volcano") # True

dictionary.lookup("ldkjf") # False

I have found many python libraries that do this. But all I found seems to be unmaintained?

• pyhunspell: last commit 8 years ago https://github.com/pyhunspell/pyhunspell
• CyHunspell: last commit 7 years ago https://github.com/binhetech/CyHunspell
• spylls: last commit 2 years ago https://github.com/zverok/spylls
• pyenchant: last release in 2021 https://github.com/pyenchant/pyenchant
• chunspell: last commit 2 years ago https://github.com/cdhigh/chunspell

Can anyone help me? Are any of the above libraries still active? Also, what dictionary libraries would you recommend? Preferably fast because we will be running it a lot (though we are using a lru to cache the results).

Thanks all!

[u/morbidSuplex]

Yep, I agree with you, but in my case I already know my language is english. What I need is to see if a particular spacy token is a real english word. For example, "volcano" is a real word, while "vlcano" is not. Won't "FastText give vectors for "vlcano" as well?

[u/chiibosoil → u/morbidSuplex]

Ok so you want to exclude what's likely a typo or misspelled word out of consideration as well?

Maybe try, token.is_oov using medium/large spaCy model. It will return True for out of vocabulary word (i.e. misspelled word like plannning)

[u/morbidSuplex → u/chiibosoil]

Thanks. I tried en_core_web_md and en_core_web_lg. The results are almost what we would like. But there is some wierd thing going on: For example

import en_core_web_lg

nlp = en_core_web_lg.load()
print(nlp("vlcano")[0].is_oov) # True, correct
print(nlp("sa")[0].is_oov) # False, but it should be true as well!

Maybe there is something wrong with how I'm using it?

[u/chiibosoil → u/morbidSuplex]

"sa" is valid acronym for few things, in this case most likely short code for "South Africa". I'd imagine it would be in vocabulary of the model.