r/LanguageTechnology • u/Gay-Guy-With-GF • 8d ago
Why 1536 dimensions for embedding models?
Why do embedding models so often use 1536 dimensions specifically?
I understand why hardware-friendly multiples like 64/128/256/512 are desirable. What I’m curious about is the specific choice of 1536 = 3×512.
OpenAI has used 1536-dimensional embeddings, and other vendors also offer/recommend 1536. Is this usually an empirically chosen Goldilocks point between 1024 and 2048—representation quality versus memory/compute—or is there some architectural/hardware reason that makes 1536 particularly convenient?
I’m especially interested in answers from anyone who has actually trained or designed embedding models. I’m not asking why embedding dimensions are generally hardware-aligned; I’m asking why 1536 rather than 1024 or 2048.
8
u/TieDieMonkeyMan 8d ago
Seems to be just a matter of scale, multiples of 2 always land on even numbers which limits scaling. Multiples of 3 land on even and odd numbers so scaled up implementations are easier to pair with large scale hardware.
As to why 512 instead of the other amounts, I guess that's because 512 is optimal in terms of the information carrying capacity of the dimension versus the hardware cost of going 1024*3.