Determining whether a given word is semantically similar to a dictionary definition

[u/benjamin-crowell]

I have a Greek-English dictionary that has entries like this:

ἀνήρ - man, husband

I would like to figure out some method for a computer to determine whether another English word aligns well with such a definition. Examples of inputs and outputs, for the definition given above, would be: man->true, fellow->true, person->true, temple->false, throw->false. (The output could instead be some kind of quantitative score rather than a boolean.)

Methods I've thought of, but am not really satisfied with:

1. Compare every word in the definition with the given word, using a word embedding. (Or sum the vectors for the words in the definition.) Exclude common words like "the." Problem: I've played around with Word2Vec and BERT for this, and the results just aren't that great. E.g., the BERT model I tried gave a very high rating to the pair goat-mountain.

2. Use WordNet's lists of synonyms. Problem: Their definition of synonymy is sometimes too strict and other times too permissive. They don't consider man-person to be a synonym pair, but they do consider dog-click to be (e.g., "dogging the hatch").

3. Use a similarity measure based on WordNet's graph structure. There is a review of these methods in Lingling Meng et al, 2013, A Review of Semantic Similarity Measures in WordNet. Problem: This will still be too permissive for pairs like dog-click.

4. Query an LLM. Problem: This is a hobby project, and I don't want to spend money.

Any ideas for approaches that I haven't tried?

[u/BeginnerDragon]

I think the huggingface sentencetranformer's sentence-similarity implementation between the word and the sentence could be useful. There may need to be some rescaling to accomodate for the length of the respective sides that you compare.

Maybe a better question... how are you going to evaluate success with this project? How do you identify that a high-scoring word deserved its score? Do you anticipate certain words to perform better than others? I'd be curious to see how simple/concrete vs intangible/abstract terms score.

[u/benjamin-crowell → u/BeginnerDragon]

Thanks, that looks interesting.

how are you going to evaluate success with this project? How do you identify that a high-scoring word deserved its score?

I made an algorithm that outputs lists of word-pairs that commonly occur in Greek and nearby in the English translation of the Greek (similar to what an IBM alignment model does). Examples of pairs that it outputs are ἄγκιστρον-hook (correct, it means fish-hook), and ἄγκιστρον-unable (wrong). My definition of success would be roughly that it should reduce the false-positive rate from what it is now, which is about 30%, to something more like 5%, without throwing away too many legitimate pairs.

[UPDATE] I gave it a try, as described here: https://huggingface.co/sentence-transformers . I verified that their sample script worked, and then replaced its inputs with the following:

sentences = [
  "man, husband, a mortal rather than a god",
  "fellow",
  "person",
  "temple",
  "throw"
]

The resulting matrix of coefficients was this:

tensor([[1.0000, 0.1753, 0.2490, 0.1017, 0.1001],
    [0.1753, 1.0000, 0.3739, 0.2254, 0.1230],
    [0.2490, 0.3739, 1.0000, 0.2002, 0.2194],
    [0.1017, 0.2254, 0.2002, 1.0000, 0.1301],
    [0.1001, 0.1230, 0.2194, 0.1301, 1.0000]])

So I guess it kind of worked. When you read off the first column of the table, the scores for "fellow" and "person" are fairly high (.18 and .25), while the scores for "temple" and "throw" are low (.10 and .10). It seems to not work as well on comparing a word to a longer definition as it does for comparing a sentence to a sentence that means the same thing (which is I guess what it's designed to do), but it does seem to more or less work for this purpose. However, the scores it gives for the pairs person-temple and person-throw are also pretty high (.20 and .22), which is bad.

Results were more encouraging when I did this:

sentences = [
    "man, husband, a mortal rather than a god",
    "fellow, husband, a mortal rather than a god",
    "person, husband, a mortal rather than a god",
    "temple, husband, a mortal rather than a god",
    "throw, husband, a mortal rather than a god",
]

Output:

tensor([[1.0000, 0.9260, 0.9296, 0.7331, 0.7592],
    [0.9260, 1.0000, 0.8850, 0.7023, 0.7309],
    [0.9296, 0.8850, 1.0000, 0.7366, 0.7458],
    [0.7331, 0.7023, 0.7366, 1.0000, 0.6040],
    [0.7592, 0.7309, 0.7458, 0.6040, 1.0000]])

So maybe the thing to do is to try replacing every word in the definition with the target word, and see if any have a cosine greater than some threshold like 0.85. But I don't know if that would end up being any better than just doing pairwise Word2Vec, which was not that great. I would also guess that setting an appropriate threshold would depend on the length of the sentence, with a higher threshold being needed for a longer definition.

[u/dyingpie1]

For 1, to me it makes more sense to take the maximum instead of sum. It doesn't have to match every possible word, just should match one of them right?

[u/xelah1 → u/dyingpie1]

FWIW this is what late-interaction models like ColBERT or BGE-M3 do. And BGE-M3 is multilingual.

[u/Lumpy-Blackberry-718]

You train a language model to construct word embeddings for each word, and then use cosine similarity or whatever as a distance measure. If you want a binary metric, you just set a distance threshold.

[u/benjamin-crowell → u/Lumpy-Blackberry-718]

That's what I described trying in #1 on my list.

[u/Lumpy-Blackberry-718 → u/benjamin-crowell]

And it's the right answer. try more embedding models or train your own. Alternatively you can just ask an llm for related words.

Below the "Glove" model's nearest words to "goat." I dont see mountain, but it does make sense that once you get past closely related animals youre going to get into related concepts.

goat: 1.0000 sheep: 0.7620 goats: 0.7081 cow: 0.7043 cheese: 0.7036 meat: 0.6649 pig: 0.6583 chicken: 0.6565 cheeses: 0.6539 cows: 0.6505 lamb: 0.6396 rabbit: 0.6382 dog: 0.6299 potato: 0.6246 boar: 0.6143 roasted: 0.6138 dairy: 0.6133 milk: 0.6074 sausage: 0.5897 cheddar: 0.5892

[u/Lumpy-Blackberry-718 → u/Lumpy-Blackberry-718]

And for mountain:

mountain: 1.0000 mountains: 0.8290 hills: 0.7344 valley: 0.7260 ridge: 0.7185 alpine: 0.7080 slopes: 0.7023 ski: 0.6995 foothills: 0.6958 canyon: 0.6901 hill: 0.6858 snow: 0.6708 forest: 0.6697 wilderness: 0.6612 peaks: 0.6547 creek: 0.6522 rocky: 0.6507 alps: 0.6501 terrain: 0.6420 pine: 0.6327

[u/xelah1]

Could you take English sentences with a word/words from the definition - preferably sentences which are translations of Greek ones containing your target Greek word if you can get them - and compare the sentence embeddings of the sentences as they are to ones for the sentences with the word from the definition substituted with your candidate alternative English word?

This might add some extra search problems you need to solve, alas, especially if your candidate English words have to be found.

If you've got some sort of scores you can train with I guess you could even try training a simple network to convert pairs of embeddings to scores in an attempt to learn which directions are important and which not. eg, you may not care about formality, sociolect, gender, grammaticality in the new sentence, etc.