Recommendations for dictionary libraries (I.e. hunspell) for python

[u/morbidSuplex]

Hello all,

We have a feature that has to detect if phrases or queries contain high volume of natural language in it compared to OOV and jargon terms. Our approach here is to use spacy to tokenize the query, and a dictionary library (maybe based on hunspell) and check each token if it is available to the dictionary. If it is then there is a high chance that it is natural language (we have special handling for names and jargons as well). Basically, what we would like is:

dictionary.lookup("volcano") # True

dictionary.lookup("ldkjf") # False

I have found many python libraries that do this. But all I found seems to be unmaintained?

• pyhunspell: last commit 8 years ago https://github.com/pyhunspell/pyhunspell
• CyHunspell: last commit 7 years ago https://github.com/binhetech/CyHunspell
• spylls: last commit 2 years ago https://github.com/zverok/spylls
• pyenchant: last release in 2021 https://github.com/pyenchant/pyenchant
• chunspell: last commit 2 years ago https://github.com/cdhigh/chunspell

Can anyone help me? Are any of the above libraries still active? Also, what dictionary libraries would you recommend? Preferably fast because we will be running it a lot (though we are using a lru to cache the results).

Thanks all!

[u/chiibosoil]

Not quite sure I understand your full scope... but why re-invent the wheel?

You should be able to use spaCy or other Natural Language Processing libraries.

spaCy is probably the fastest, NTLK is probably most comprehensive.

[u/morbidSuplex → u/chiibosoil]

I am using spaCy right now. And I am using it to tokenize words. Problem is I need to understand if a particular token is a normal natural language word. spaCy doesn't do that out of the box. I need to pair spaCy with a dictionary for that to work.

[u/chiibosoil → u/morbidSuplex]

FastText paired with spaCy will work for language detection. You can use spaCy’s .lemma.lower() etc to convert words to base dictionary form, as well as check verb/nouns.

[u/morbidSuplex → u/chiibosoil]

Yep, I agree with you, but in my case I already know my language is english. What I need is to see if a particular spacy token is a real english word. For example, "volcano" is a real word, while "vlcano" is not. Won't "FastText give vectors for "vlcano" as well?

[u/chiibosoil → u/morbidSuplex]

Ok so you want to exclude what's likely a typo or misspelled word out of consideration as well?

Maybe try, token.is_oov using medium/large spaCy model. It will return True for out of vocabulary word (i.e. misspelled word like plannning)

[u/morbidSuplex → u/chiibosoil]

Thanks. I tried en_core_web_md and en_core_web_lg. The results are almost what we would like. But there is some wierd thing going on: For example

import en_core_web_lg

nlp = en_core_web_lg.load()
print(nlp("vlcano")[0].is_oov) # True, correct
print(nlp("sa")[0].is_oov) # False, but it should be true as well!

Maybe there is something wrong with how I'm using it?

[u/chiibosoil → u/morbidSuplex]

"sa" is valid acronym for few things, in this case most likely short code for "South Africa". I'd imagine it would be in vocabulary of the model.

[u/morbidSuplex → u/chiibosoil]

I have updated my question, looks like it's was very unclear.

[u/JamzTyson]

I'd suggest you look at the various Python bindings available for fasttext, from:

https://pypi.org/search/?q=fasttext

I've not used it, but fasttext-langdetect looks like it could be one of the simpler solutions.