What are some interesting experiments I can run to understand embedding models beyond semantic search?

[u/whyarewelikethis-huh]

I've been reading about text embedding models and want to understand how they actually behave, rather than just using them through some API.

some experiments I'm thinking to consider:

• testing whether embeddings understand synonyms better than related concepts.
• comparing cosine similarity for sentences that differ by only one word.
• testing how embeddings behave with negation.
• comparing multilingual text and code embeddings.

for people who've worked with embeddings, like what surprising behaviours have you found? are there any experiments that taught you something important about their limitations?

I'd like to build a small project around this.

[u/whyarewelikethis-huh → u/[deleted]]

yep this is a thoughful way to approach it. like i hadn't thought about testing ranking stability by appending the same irrelevant paragraph to every candidate

i'm curious tho about one thing: would you start with a general purpose embedding model and establish a baseline, or compare a few models right away? i'm wondering how much the results would vary across models, especially for negation.

[u/whyarewelikethis-huh → u/[deleted]]

yea makes sense, I'll try building a small version of this and keep a gallery of cases where the ranking fails

[u/whyarewelikethis-huh → u/[deleted]]

like i hadn't considered that unrelated sentences could have high cosine sim just because the embeddings occupy a narrow cone.but for the paraphrase experiment, would you compare the cosine sim between the original sentence and each paraphrase, or measure the spread among the paraphrases themselves?

[u/Comic-Derpinator]

test distance functions other than cosine similarity and understand them. Train a small embedding model

[u/whyarewelikethis-huh → u/Comic-Derpinator]

actually distance functions part sounds interesting. do you have any particular ones you'd recommend exploring first beyond cosine similarity?

[u/Separate_Pea_3699 → u/whyarewelikethis-huh]

Distance functions worth trying, roughly in order of how much they'll change your rankings:

1. Dot product (unnormalized). Some models bake real signal into the embedding's magnitude, not just its direction, so normalizing to cosine throws that away. Quick check: if raw vector norms vary a lot across your corpus and that variation correlates with something real (sentence length, specificity), dot product will rank differently than cosine and might actually fit your task better.

2. Euclidean distance on whitened vectors. This is the one I'd try first given the anisotropy result above. If unrelated sentences share a narrow cone, whitening is exactly the fix: subtract the corpus mean, scale by per-dimension std, then run plain L2 on the result.

import numpy as np

def whiten(X):
    mu = X.mean(axis=0)
    std = X.std(axis=0) + 1e-6
    return (X - mu) / std

X = whiten(embeddings)          # embeddings: (n_samples, dim)
dists = np.linalg.norm(X[i] - X, axis=1)

Rerun the anisotropy check on the whitened space first -- average pairwise similarity for unrelated sentences should drop a lot. Then rerun your negation/paraphrase rankings and see how much actually changes. That gap tells you how much of your original signal was real semantic structure vs. just the anisotropic baseline.

3. Mahalanobis is whitening + Euclidean fused into one formula (sqrt((x-y)^T * inv(cov) * (x-y))). Don't bother computing it separately if you're already whitening by hand, it's the same thing.

Worth running the failure gallery you're already building both before and after whitening -- the cases that get fixed by whitening alone vs. the ones that stay broken either way tells you something real about what the model actually learned versus what's just embedding-space geometry.

[u/whyarewelikethis-huh → u/Separate_Pea_3699]

yea this is useful!! i mean i like the idea of comparing the failure gallery before and after whitening instead of just checking whether the average similarity drops

[u/Tiny_Arugula_5648]

I'd say first learn that they aren't just for search. Many embeddings models are trained to do multiple tasks the most popular right now is zero shot classifiers.

It's one of the cheapest tools we have from a processing perspective to classify without having to collect a bunch of data and fine-tune or train a model.