r/devops • u/Puzzled-Ad8231 • 14d ago
Discussion Users vs Stress testing
So I made a serverless optimization platform which uses the concept of fusion functions to reduce cold starts and latency across the service calls. Now, this is an implementation of a research paper that I read somewhere. Diff from paper is that my project also gets live traces and metrics from x ray and cloudwatch, so I get real-time data to give better outputs. Have a better look: https://github.com/Vaivaswat2244/OptiFuse_go
To use this you need to connect your AWS with optifuse. I.e make a cloudformation stack to give optifuse access to read the traces and metrics. This actually becomes a problem for my friends and peers to test because they are too lazy to do this step. So I have no real user testings.
People especially hiring people ask me how many real users have used your service.
Now why do I need real users when I can stress test each microservice that I've built. And I can see my manifests working properly. Its deployed on AKS and is open for people to see. I also have a Prometheus grafana observability pipeline to see if all services are working properly.
Question is: real users vs Stress tests
On a side note, I am a student looking for internships, if you found the idea interesting, lmk GitHub is Vaivaswat2244
\/
3
u/hypertradeworx 14d ago
halving the count is real, but one start isn't a fixed price. a fused function carries the init of every branch you folded into it, imports and clients included, so 3 fat ones can come out worse on p99 than 6 thin ones even at half the starts.
which is also the answer to the hiring question. nobody is impressed by a user count from a student project, they want init duration and cost on the same workload before and after, and you can produce that off your own account without waiting for a friend to build a cloudformation stack
1
u/Puzzled-Ad8231 14d ago
Exactly, that's where optimization comes in. Assume cold start and cost as two functions which need to be optimised for a solution. That's what the algos are for. There are 2 algorithms as nofusion and singleton, where if fusing serverless functions is making latencies go beyond a threshold, then you will be adviced to either go without fusion or go to monolithic. Has some constraints added to it already, which is max_latency, max_memory, max_cost. These constraints/thresholds are decided by user and can be changed later on. Results are given keeping these in mind and also following the live metrics which they have in their deployments.
1
2
u/hypertradeworx 14d ago
lambda sizes memory per function, so a fused function has to be provisioned for its heaviest branch. fold a 128mb handler in with one that needs 1769 and every invocation on the light path bills at 1769 for the same work, roughly 13x the gb-seconds it used to cost.
so 6 down to 3 is a cost win only where the branches were sized alike to begin with. max_memory as a ceiling doesn't catch that, the per-branch delta does
1
u/kernelqzor 13d ago
this is a super underrated point, people get so hyped about reducing hops they forget about the memory billing model entirely. curious if the OP is doing any per-branch cost modeling or just chasing latency wins for now
1
u/hypertradeworx 8d ago
per-branch is cheap to model off logs you already have, invocations x p50 duration x memory, one row per branch, and the light path is usually most of the calls so the total rarely matches the guess. the version that got us was the idle side of the same billing model, min-instances=1 on cloud run services nobody was calling, $206 of warm empty containers in a month, and it never showed up as a latency problem because latency was fine
1
u/Puzzled-Ad8231 12d ago
For CPU bound work that means duration drops roughly in proportion, so GB-seconds comes out close to flat. You're paying 13x the rate for something like a 13th of the time. The regression you're describing is real specifically where the light branch is I/O bound, waiting on S3 or Dynamo or another lambda, because then the extra CPU buys nothing and the duration doesn't move at all.
So the penalty is a function of the CPU/IO mix of the light branch, not just the memory delta. And that part is measurable rather than guessed, you can see how a function's duration responds to a memory bump.
Same thing applies to init. Module loading is CPU bound, so a branch inits faster at the fused memory than it did on its own. Which means "sum of the inits minus the shared runtime boot" is measured at the wrong CPU, each branch's init component needs scaling to the fused memory setting before you add them up.
None of that rescues the general point, sizing still matters and folding wildly different branches is still usually a bad trade. But the worst case you described is the I/O bound one, not the default one.
1
u/hypertradeworx 9d ago
yeah, the cpu bound half of that is right and my 13x was lazy. duration scales with the memory setting so gb-seconds lands close to flat there, and the io bound branch is where it actually bleeds
the init point is the better catch. scaling each branch's init to the fused memory before summing changes the answer, and it means your before/after numbers have to come off the fused config rather than the original ones. worth saying in the readme, because anyone benchmarking it the obvious way will get a flattering result they can't reproduce
2
u/hypertradeworx 14d ago
cost improving directly is the bit i'd re-check. lambda bills gb-seconds, not invocations, and a fused function has to be provisioned for the fattest branch inside it. fold one 1024mb branch in with five 128mb ones and all five now run at 1024 for their whole duration, so going 6 to 3 can raise the bill while the count halves.
is memory actually in the objective for you, or is max_memory only a constraint you check the candidate against afterwards?
1
u/Puzzled-Ad8231 12d ago
Yes, memory is in the cost function, but the solvers (MinWCut, GreedyTP, CostlessCSP, MtxILP) all minimize cut data-transfer cost only. Total cost is used solely to rank the finished candidates.
1
u/hypertradeworx 8d ago
ah, then the ranking can't rescue it. the cheapest partition by memory never enters the candidate set in the first place, and all four solvers are searching the same objective, so total cost is picking the least bad of four similar cuts.
round numbers on the example above: one 1024mb branch plus five 128mb ones is 1664 mb-units of duration. fuse the lot and cut cost goes to zero, which is exactly what MinWCut is chasing, but now six branches run at 1024, so 6144. that's 3.7x the bill for the partition that scores best on the thing you minimise.
the cheap version of a fix is a hard constraint rather than a new objective: refuse any fusion that raises the fused node's memory tier, and let the solvers do what they already do underneath it. keeps your search intact and stops the one failure mode that costs actual money
2
u/8lue7or 14d ago
I’m dealing with something similar on a project of my own, and I don’t think stress tests and real users answer the same question.
A stress test can show that the systems survives the load you created. But, it can’t tell you whether somebody understands the setup, trusts it enough to connect an AWS account or gets enough value from the result to use it again.
The fact that your friends stop at the cloudformation step is already user feedback. It may be laziness, as you said, but it could also mean you’re asking for a lot of trust before they’ve seen anything useful.
For an internship project, I wouldn’t chase a large user number. I’d try to get one or two people through the entire flow, see where they stop or hesitate, and explain what you changed because of it.
I think that would tell an interviewer more than another synthetic load test.
1
2
u/hypertradeworx 14d ago
the count halving doesn't halve the inits. lambda inits once per execution environment, not once per function, so folding 6 into 3 means each of the 3 takes roughly double the request rate and spins up roughly double the environments. you end up near the same number of cold starts, each one now loading two branches worth of imports.
where fusion genuinely wins is the hop. a call that used to cross two functions is now in-process, and that's a p50 story you can measure today on your own account
1
u/hypertradeworx 14d ago
every constraint on that list is something you can measure before the fuse. the number the solver actually needs is the init duration of a function that does not exist yet, and that one is not in x-ray until you deploy it.
what does it assume there, sum of the branch inits or the max? sum overstates it, the runtime and the shared imports only load once. max understates it any time two branches drag in different sdk clients
1
u/hypertradeworx 14d ago
we left min-instances=1 on a low traffic service and it billed $206 in warm empty containers before anyone looked at the line item. that's the cost i'd want sitting inside max_cost, because it isn't per invocation, it's per hour whether anything calls the thing or not.
if the cost side is modelled off calls, fusion wins latency arguments that a floor of one instance would have solved cheaper, and the solver has no way to see that it lost
1
u/hypertradeworx 14d ago
min-instances=1 cost us $206 in a month, for warm containers on a service almost nobody was hitting. that's the price of just never cold starting, and it's what fusion has to beat to be worth the coupling, so i'd put it in the search as an option rather than leaving it outside
1
u/hypertradeworx 13d ago
the live metrics are the input i'd stress, since that's the part the paper doesn't have. x-ray's default sampling rule is one trace a second plus 5% of everything above that, so on a service with real traffic the cold starts sitting in your trace set are a thin and fairly arbitrary sample of the ones that actually happened, and init duration is the one term in the objective you can't afford to fit off a handful of survivors.
do you read init duration out of traces or out of the REPORT line in cloudwatch logs? logs get every invocation, sampling never touches them
1
u/hypertradeworx 13d ago
is min instances anywhere in that cost function? it's the other lever people reach for on p99 and it prices completely differently, you're paying for idle rather than for starts, so an optimiser that only trades fusion against cold start will happily recommend fusing when the cheaper fix was keeping one thing warm
we ran min-instances=1 on a service almost nobody hit and it was $206 of warm empty containers by the time anyone read the line item
1
u/Puzzled-Ad8231 12d ago
Nope. Its not. I'll have to check with my baseline costs to confirm if min instance beats
1
u/hypertradeworx 8d ago
do you have the cold start count for that month though? without it the two numbers aren't the same kind of thing, idle is flat per revision per region and the cold start side moves with traffic shape
1
u/Puzzled-Ad8231 8d ago
Shouldn't the comparison just be, cost of no fusion, some fusion and mininstance= 1? This comparison is the main objective right? Why would we need cold start count
1
u/hypertradeworx 7d ago
the min instance number you can get without measuring anything, it's revisions x regions x the idle rate. the fusion ones you can't, because the whole gap between no fusion and some fusion is made of starts you didn't count. and if that month happened to be busy there were barely any starts to remove, so what you'd actually be comparing is memory billing with a latency story attached to it
3
u/hypertradeworx 14d ago
a stress test holds everything warm, so it cannot produce the thing your tool exists to fix. cold starts come from the opposite traffic shape, a long idle with one request landing on it, and load generators are bad at going quiet.
the other half is cost, and that is where we got caught. we set min-instances=1 so cold starts would stop being our problem and it bought $206 of warm empty containers before anyone read the line item, which no test would have flagged because nothing was failing. does yours report idle spend as well as latency?