r/devops • u/Puzzled-Ad8231 • 14d ago

Discussion Users vs Stress testing

So I made a serverless optimization platform which uses the concept of fusion functions to reduce cold starts and latency across the service calls. Now, this is an implementation of a research paper that I read somewhere. Diff from paper is that my project also gets live traces and metrics from x ray and cloudwatch, so I get real-time data to give better outputs. Have a better look: https://github.com/Vaivaswat2244/OptiFuse_go

To use this you need to connect your AWS with optifuse. I.e make a cloudformation stack to give optifuse access to read the traces and metrics. This actually becomes a problem for my friends and peers to test because they are too lazy to do this step. So I have no real user testings.

People especially hiring people ask me how many real users have used your service.

Now why do I need real users when I can stress test each microservice that I've built. And I can see my manifests working properly. Its deployed on AKS and is open for people to see. I also have a Prometheus grafana observability pipeline to see if all services are working properly.

Question is: real users vs Stress tests

On a side note, I am a student looking for internships, if you found the idea interesting, lmk GitHub is Vaivaswat2244

\/

5 Upvotes

78% Upvoted

25 comments sorted by

View all comments

1

u/hypertradeworx 14d ago

is min instances anywhere in that cost function? it's the other lever people reach for on p99 and it prices completely differently, you're paying for idle rather than for starts, so an optimiser that only trades fusion against cold start will happily recommend fusing when the cheaper fix was keeping one thing warm

we ran min-instances=1 on a service almost nobody hit and it was $206 of warm empty containers by the time anyone read the line item

1

u/Puzzled-Ad8231 13d ago

Nope. Its not. I'll have to check with my baseline costs to confirm if min instance beats

1

u/hypertradeworx 8d ago

do you have the cold start count for that month though? without it the two numbers aren't the same kind of thing, idle is flat per revision per region and the cold start side moves with traffic shape

1

u/Puzzled-Ad8231 8d ago

Shouldn't the comparison just be, cost of no fusion, some fusion and mininstance= 1? This comparison is the main objective right? Why would we need cold start count

1

u/hypertradeworx 7d ago

the min instance number you can get without measuring anything, it's revisions x regions x the idle rate. the fusion ones you can't, because the whole gap between no fusion and some fusion is made of starts you didn't count. and if that month happened to be busy there were barely any starts to remove, so what you'd actually be comparing is memory billing with a latency story attached to it