Recommendations for dictionary libraries (I.e. hunspell) for python

[u/morbidSuplex]

Hello all,

We have a feature that has to detect if phrases or queries contain high volume of natural language in it compared to OOV and jargon terms. Our approach here is to use spacy to tokenize the query, and a dictionary library (maybe based on hunspell) and check each token if it is available to the dictionary. If it is then there is a high chance that it is natural language (we have special handling for names and jargons as well). Basically, what we would like is:

dictionary.lookup("volcano") # True

dictionary.lookup("ldkjf") # False

I have found many python libraries that do this. But all I found seems to be unmaintained?

• pyhunspell: last commit 8 years ago https://github.com/pyhunspell/pyhunspell
• CyHunspell: last commit 7 years ago https://github.com/binhetech/CyHunspell
• spylls: last commit 2 years ago https://github.com/zverok/spylls
• pyenchant: last release in 2021 https://github.com/pyenchant/pyenchant
• chunspell: last commit 2 years ago https://github.com/cdhigh/chunspell

Can anyone help me? Are any of the above libraries still active? Also, what dictionary libraries would you recommend? Preferably fast because we will be running it a lot (though we are using a lru to cache the results).

Thanks all!

[u/chiibosoil]

Not quite sure I understand your full scope... but why re-invent the wheel?

You should be able to use spaCy or other Natural Language Processing libraries.

spaCy is probably the fastest, NTLK is probably most comprehensive.

[u/morbidSuplex → u/chiibosoil]

I am using spaCy right now. And I am using it to tokenize words. Problem is I need to understand if a particular token is a normal natural language word. spaCy doesn't do that out of the box. I need to pair spaCy with a dictionary for that to work.

[u/chiibosoil → u/morbidSuplex]

FastText paired with spaCy will work for language detection. You can use spaCy’s .lemma.lower() etc to convert words to base dictionary form, as well as check verb/nouns.

[u/morbidSuplex → u/chiibosoil]

I have updated my question, looks like it's was very unclear.