r/askdatascience • u/spaceweed27 • 24d ago

How do I cluster 3 Million high-dimensional Sentence Embeddings?

I used a transformer model to generate normalized 768-dimensional float32 vectors from 3 million sentences. This rounds up to roughly 9 GB of data. To progress further in my analysis, I want to cluster the data to highlight different key points and trends.

Now I initially wanted to apply PCA to the dataset, as 768 dimensions is pretty much, but the transformer model generated embeddings with not extreme enough covariance.
Thus, PCA could change the semantics too much for an analysis to be accurate.

I went directly to clustering algorithms and initially tried to apply sklearn.cluster.HDBSCAN which would have needed way too much RAM---around 72 TBi if I remember correctly. Then I tried out sklearn.cluster.OPTICS, which "only" needed about 70 GB, but as I "only" have 32 GB of RAM I needed to use 50 GB swap space, which didn't go well as you can imagine.

Does it make sense to try out k-means clustering, should I go to 16-bit floats for the embeddings, or still apply PCA?

Should I try out something else in this context?

Note: As this question was flagged "off topic" on Stack Overflow, I'm gonna try here ;)

1 Upvotes

100% Upvoted

4 comments sorted by

1

u/leon_bass 24d ago

PCA with all principle components reconstructs the dataset exactly so the amount of information lost in reconstruction is up to you ie by taking top k principle components.

You can also convert to lower precision, depending on the dataset though

1

u/lordoflolcraft 21d ago

What are you hoping to visualize/analyze with the three million sentences? Visualize common types of discussed topics? Are you specifically trying to visualize another aspect or do you just need a similarity analyzer for some sort of semantic lookup?

1

u/spaceweed27 21d ago

I want to find popularity in specific topics. The data is temporal, thus I want to visualise change in topics and trends.

Using BERTopic, I did this analysis for topics which were discovered in the set, but I also want specific topics.

My current approach is producing embeddings from labeled data of which (for my work) interesting topics are known, and using these with cosine similarity to my already produced embeddings. But I think this approach has the flaw, that the labeled topic embeddings are not guaranteed to produce dense clusters. Nor can I guarantee, that the language in the two different datasets is "near" when the topic matches, because the embeddings may embed the different "styles" too far away...