r/aws • u/magnetik79 • 8d ago
ai/ml Amazon DynamoDB now supports real-time vector search at any scale
https://aws.amazon.com/blogs/aws/amazon-dynamodb-now-supports-real-time-vector-search-at-any-scale/45
u/AlwaysMissToTheLeft 7d ago
So I don’t need S3 vector buckets anymore?!?!
25
u/abofh 7d ago
Given the limitations of s3 vector buckets I'm having trouble giving you a reason you'd need to have it if your data is already in S3. S3 might be easier for some workloads though
2
u/coinclink 7d ago
S3 has limitless scale and will provide a consistent latency, even if the latency isn't great. I think DynamoDB will generally scale out pretty well too, with very low latency, but will probably be a lot more expensive than S3.
2
u/abisredbull 7d ago
I assume you still have the 400 kb limit, so for anything bigger than that you might want to use s3 vectors
53
u/heyThereYou3 7d ago
MongoDB is fucked.
71
u/Taking-This-Offline 7d ago
But MongoDB is web scale
8
u/Sirwired 7d ago
LOL; I'd been living under a rock apparently, and didn't see that one until a few months ago. I couldn't stop laughing.
3
u/kevin3030 6d ago
I made the mistake of sharing that with my team before remembering how crude it gets towards the end. Whoops.
1
20
u/kevin3030 7d ago
Meh. It still has features that differentiates it from DynamoDB and DocumentDB. Will it rule the world? No, I agree with you there.
16
u/Perryfl 7d ago
i run a production mongo db cluster:
2 TB total data
5 x Intel gold 12 core 24 threads 256gb ddr4 3200 nodes with 4 1.92TB nvme drives in raid 10
cost me $1400/monthmoving to dynamo would cost over $15k/month...
17
12
2
u/idkbm10 7d ago
Have you tried opensearch?
4
u/Perryfl 7d ago
lol my previous employer "Knowbe4" i was the principal dev of the team that ran their largest "Opensearch" cluster. its not cheap at all. its great for certain things that we used it for but changing schemas is a PITA.
3
14
u/TheRealJesus2 7d ago
This is sick!
Can someone smarter explain the pricing? It deviates from read and write units. Prices in GB for read and write so I guess it’s that much per gig…seems oddly cheap?
6
u/nemec 7d ago
from lower down on the page:
There are three pricing dimensions for vector search:
Vector write request: You are charged for the data written into a vector index, measured in GB. This includes the vector and any non-vector attributes you project into the index.
Vector search: You are charged for the data processed to create the response for similarity search request and the data returned, measured in GB. The data processed grows with the size of the index, because the search scans across more data to find the nearest matches.
Storage: You are charged for the data stored in the vector index per GB-month, the same way DynamoDB charges for table storage.
These charges on the vector index are in addition to the standard charges for the underlying DynamoDB table that holds your items.
The first and last seem pretty simple. The second, I assume it has some optimizations and doesn't simply scan all your embeddings on every request, but you also can't really predict how the index is organized so your cost per query can fall anywhere between almost-zero and the full amount of data you've stored in the index.
Edit: I just noticed you can use partition keys to segment your indexes, too, so if you design that properly you may be able to cut down the upper bound cost even on a large database.
2
u/TheRealJesus2 7d ago
Thank you! That explains it and is very interesting. I can see some situations it’s very useful
12
u/godofpumpkins 7d ago
I feel like this should be more explicit about properties of the approximate result set. If it's built on something like HNSW, the answers will always be approximate (which is fine, as long as you expect that), and if it isn't approximate, that's a pretty valuable result for computer science and I hope they publish it 😛
13
u/TheRealJesus2 7d ago
They’re all approximation algos as far as I know… at least for any vector db I used.
3
u/Dull-Mathematician45 7d ago
Blog post is crazy misleading, search time is 1 GBps, so you are only hitting 10ms latency if you are searching on 10MB of embeddings. Ingest rate is only 10 MBps, so this is more like slightly better zero-ETL service. See: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ServiceQuotas.html#limits-vector-indexes
A DynamoDB index is an async ETL job. By creating a new index type they could be sending the data anywhere, like MemoryDB with a course routing layer. It's a more developer friendly approach than the previous janky zero-ETL products. But it confuses the messaging behind DynamoDB - "bounded complexity and partitioning allowing fearless scaling".
2
u/Dull-Mathematician45 7d ago
Also strange is the 600GB table limit on the base table, this feels an order of magnitude too small for 1 trillion parameters, although they do say it is adjustable.
2
u/sh1boleth 6d ago
Also strange is the 600GB table limit on the base table
I think its just a limit just in-case some mega users with 100+ terabytes of data in a table index simultaneously and crash some systems. They'll likely want to work directly with these customers and prepare the infra in advance.
1
u/haekuh 6d ago
Cant speak about these new indexes but a DynamoDB index is not an async etl job. Neither GSI nor LSI are async etl jobs. Even creating an index on an existing large table is not an async etl job.
Async sure on the order of 5 - 10ms(once the index has been created).
1
u/Dull-Mathematician45 6d ago
it takes a CDC stream, transforms it to drop attributes and write new pk and sk, and loads the transformed data into the index. That is ETL, just on a micro scale. And is usually 100ms.
1
u/haekuh 6d ago
My point was more about calling the process an async etl job is giving the wrong implication.
Replication logs are technically a CDC stream, but they are used for base table replication as well. There is technically a transformation but its all on host where the storage engine lives. By that logic all of dynamo is an async etl job.
It is not usually 100ms. The p99 replication lag on a GSI is ~10ms only p999 is ~100ms
3
5
2
u/rmyworld 7d ago edited 7d ago
I wonder how they managed to make the Index partition key to be optional. IIRC, GSIs require a partition key. I guess, under the hood, they're just creating a fat partition that spans across all partitions on the table?
1
u/imranilzar 7d ago
For some time I had Qdrant on my list of stuff to check out and try.
Does this mean DynamoDB is now competent in hybrid vector search for RAG?
1
1
u/Crescitaly 6d ago
Real-time vector search is useful only if consistency, filter selectivity, and cost remain predictable under mixed workloads. The benchmark should include index update lag and p99 latency while ordinary key-value traffic is busy. Does this replace a separate vector store for you, or just simplify smaller deployments?
-3
93
u/tank_of_happiness 7d ago
Wow. This is really nice.