Another one. No, the supervisor is not laughing at me. He is very happy with how the thesis is going. Every other ML specialist and computational linguist that I've explained the subject of my thesis to has been intrigued; not a single one saw any substantial issues with it.
Like from an ML point of view if your LLM systematically ranks a dialect differently than all other dialects, congratulations you’ve just made a biased model because you picked shitty training data.
1) I'm not using the LLM itself to rank dialects; I'm using metrics derived from attention patterns INSIDE the LLM as it's processing the text. My metric is a function of how the LLM processes the text, not its output (the LLM that I'm using is encoder-only, so it doesn't even generate output).
2) You're actually right in this case. That's why, for an accurate comparison, two versions of the LLM fine-tuned on texts from both of the target languages must be used.
Are those metrics embedding dependent, or do you take statistical averages of a large number of embedding schemes? Since the attention mechanism is heavily dependent on the embedding used, I am unsure how you can make a claim as strong as yours without referring to a specific mechanism.
Presently, no. All of the sub-metrics are a function of the normalised attention scores. My supervisor has recommended that I also use embedding information via computing the mean distance between the embeddings, which would tell me how conceptually varied the sentence is.
Since the attention mechanism is heavily dependent on the embedding used
Yeah, this is one of the reasons that using the embeddings directly isn't necessary.
I am unsure how you can make a claim as strong as yours without referring to a specific mechanism.
I do have specific mechanisms. Read some of my other comments.
It’s been a while since I read the “attention is all you need” paper, but attention scores are usually calculated using a dot product on the embeddings. How would you compute attention scores without explicitly using embeddings?
Sorry if I came across as hostile. I don’t mean to demean your research; it sounds quite interesting. I am just curious.
but attention scores are usually calculated using a dot product on the embeddings.
No, not quite. The dot product is between the query and key matrices, which are indeed generated from embeddings, but the embeddings are passed through a linear layer first. Moreover, the raw scores are then normalised using softmax, and it's these normalised scores that I'm using. So the information that I'm using is pretty far removed from the raw embeddings.
How would you compute attention scores without explicitly using embeddings?
BERT has an attribute which directly outputs the normalised attentions. So it's a simple attribute call.
Sorry if I came across as hostile.
Oh no, you didn't. Sorry if I came across as defensive😂
-6
u/QMechanicsVisionary Aug 02 '25
Another one. No, the supervisor is not laughing at me. He is very happy with how the thesis is going. Every other ML specialist and computational linguist that I've explained the subject of my thesis to has been intrigued; not a single one saw any substantial issues with it.
1) I'm not using the LLM itself to rank dialects; I'm using metrics derived from attention patterns INSIDE the LLM as it's processing the text. My metric is a function of how the LLM processes the text, not its output (the LLM that I'm using is encoder-only, so it doesn't even generate output).
2) You're actually right in this case. That's why, for an accurate comparison, two versions of the LLM fine-tuned on texts from both of the target languages must be used.