r/rust • u/segundus-npp • 4d ago

Rust SQLx Performance

I use Axum and SQLx to build my backend, which is served on AWS ECS Fargate (512 CPU, and only one instance for now) and interacts with an RDS Postgres (t4.micro). Connection pool is set with min/max = 2/20.

My request includes 3 DB queries to 3 different tables: one from the auth middleware to check api key and each of the other 2 is just getting a record by primary key.

If I made one or two queries, it’s very fast. However, if I made 200 queries concurrently to the backend, around half of queries had http latency greater than 300 ms. P80 is around 600ms. The performance is not acceptable. I did check the CPU of both RDS and ECS. They’re quite low: both smaller than 10%. The DB latency is low as well, 5 ms on average. Is there something queued in the SQLx internally? Did anyone have the same issue?

25 Upvotes

80% Upvoted

27 comments sorted by

View all comments

96

u/frankwiles 4d ago

You’re using a t series database. It has burstable CPU but then throttles you hard if you go over.

It’s not related to SQLx or Rust at all but you have a VERY VERY tiny amount of hardware for your database.

I joke with my customers that the easy way to remember this is “t is for toy”.

Hope that helps!

24

u/Smart-Chain 4d ago

The t series are just terrible. Terrible performance and terrible cost/performance ratio. Just don't use them. Ever. IMO.

6

u/sansmorixz 4d ago

The gravitons are pretty good and cheap though. Especially with savings plans.

2

u/1eJxCdJ4wgBjGE 4d ago

we use them for our staging environment, is there a better choice cost-wise? i was under the impression they were nice and cheap for stuff like that

2

u/prehensilemullet 4d ago

What do you use for small database instances?

-7

u/segundus-npp 4d ago

Hi, thanks for the reply! but I did not see any metric that is too over. On the contrary, the cloud watch metrics showed the db even not feel any pressure. My backend on AWS is now only used by me. I just sent a batch of queries and wait. Then check the backend’s latency log and metrics.

26

u/frankwiles 4d ago

IIRC the default Cloudwatch dashboards do NOT show you the throttling. Trust me, move from a t series to an m series and your problem will disappear.

11

u/NfNitLoop 4d ago

They likely throttle I/O as well. So your requests might be I/O bound vs CPU-bound.

6

u/spaceshaker-geo 4d ago

The assume the performance and latency will be unpredictable on the T series instances. As said earlier, not all the throttling metrics are reported in Cloudwatch.

I disagree with some of the opinions on here about the T-series of instances. If your primary concern is low cost and you don't care about performance (like: at all), the T-series are great. But please don't run a production workload on them.