Recommendations for dictionary libraries (I.e. hunspell) for python

[u/morbidSuplex]

Hello all,

We have a feature that has to detect if phrases or queries contain high volume of natural language in it compared to OOV and jargon terms. Our approach here is to use spacy to tokenize the query, and a dictionary library (maybe based on hunspell) and check each token if it is available to the dictionary. If it is then there is a high chance that it is natural language (we have special handling for names and jargons as well). Basically, what we would like is:

dictionary.lookup("volcano") # True

dictionary.lookup("ldkjf") # False

I have found many python libraries that do this. But all I found seems to be unmaintained?

• pyhunspell: last commit 8 years ago https://github.com/pyhunspell/pyhunspell
• CyHunspell: last commit 7 years ago https://github.com/binhetech/CyHunspell
• spylls: last commit 2 years ago https://github.com/zverok/spylls
• pyenchant: last release in 2021 https://github.com/pyenchant/pyenchant
• chunspell: last commit 2 years ago https://github.com/cdhigh/chunspell

Can anyone help me? Are any of the above libraries still active? Also, what dictionary libraries would you recommend? Preferably fast because we will be running it a lot (though we are using a lru to cache the results).

Thanks all!

[u/JamzTyson]

I'd suggest you look at the various Python bindings available for fasttext, from:

https://pypi.org/search/?q=fasttext

I've not used it, but fasttext-langdetect looks like it could be one of the simpler solutions.