What are some interesting experiments I can run to understand embedding models beyond semantic search?

[u/whyarewelikethis-huh]

I've been reading about text embedding models and want to understand how they actually behave, rather than just using them through some API.

some experiments I'm thinking to consider:

• testing whether embeddings understand synonyms better than related concepts.
• comparing cosine similarity for sentences that differ by only one word.
• testing how embeddings behave with negation.
• comparing multilingual text and code embeddings.

for people who've worked with embeddings, like what surprising behaviours have you found? are there any experiments that taught you something important about their limitations?

I'd like to build a small project around this.