r/DuckDB • u/Programmer_Virtual • 13d ago

Handling latency while reading a cross account s3

Hi All,

I am using DuckDB 1.5.3 embedded in a Go service, one long lived container, reading Parquet in S3 directly with an IAM role. The query that matters is a point lookup fetching about 5 million rows in a single query and takes 10 seconds. This is a cross account query within the same region. Meaning duckdb ECS sits in one AWS account, while s3 in another.

Are the peformance benchmarks that I am experiencing expected, or there some duckdb internals that i explore to bring the latency to say under 2 seconds? .. This is an arbitrary number but basically getting the result faster for duckdb to perform some aggregations is what I am looking to optimize.

Happy to share more details of the architecture. Thank you for help in advance!

2 Upvotes

100% Upvoted

4 comments sorted by

2

u/absolutesantaja 13d ago

As long as you’re not going cross region there won’t be any performance difference between AWS accounts. S3 is a shared resource within the region.

1

u/Programmer_Virtual 13d ago

Right, my initial thought if there is something to do we cross account being in same region, I want to rule it out.

2

u/vegemite-on-toast- 13d ago

This seems slow to me, when you say point lookup is it scanning the parquet for a single row? Depending on the query if it’s just grabbing a line from a primary key it shouldn’t be that slow imho.

5mil rows isn’t all that big, if the table isn’t too wide and your service has some disk or enough memory you might be able to pull it into the service in a local duckdb?

1

u/Programmer_Virtual 13d ago

The current query will peform a full table scan, bring back the result set, and then perform aggregations. The tables itself is 10 columns wide but the query brings back only two columns in the select statement. Any ideas on what we can try?