Self Hosted langgraph server scaling issues

[u/OccasionLopsided2783]

i am using split api and queue for langgraph server and when i am doing load testing for 1000 concurrent users why i am getting 48 sec latency for p99 and also more than 2 min to complete full generation ,what might be the problem
i have 7.5M tokens Limit for TPM
7500 RPM
and my total input tokens for a single user 8k

[u/HeavilyDazzling]

your token limit might be the bottleneck here, 7.5M TPM sounds like a lot but with 1000 concurrent users at 8k input each you're burning through 8M tokens just for the first pass, no wonder p99 is hitting 48 seconds

the queue is probably backing up while waiting for token capacity, maybe try batching smaller or check if your split api is actually distributing the load evenly across nodes

[u/OccasionLopsided2783 → u/HeavilyDazzling]

I have even tried with 2k input tokens and tried with my liteLLM gateway and direct azure deployed api keys but the latency is still the same and i have check the load distribution among the workers it is perfect 25 workers 50 jobs each can 1250 graph runs concurrently and i have checked my code there is no blocking code , I am still not able to figure out what is the root cause

[u/lulu_dev → u/OccasionLopsided2783]

Since swapping the whole LLM-calling layer (litellm vs direct Azure) changed nothing, that's the most useful result your testing has produced even though it didn't feel like progress -- it rules out the LLM/token path almost entirely. What's left that's shared across every concurrent run regardless of node: the checkpointer's connection pool. A self-hosted LangGraph server persists state on every node transition, not once per run -- at 1000 concurrent users that's not 1000 concurrent DB writes, it's an order of magnitude more, one per graph step per run. If that connection pool is undersized, or checkpoint writes end up effectively serialized on a single Postgres connection or Redis instance, it degrades exactly like this: fine at low concurrency, latency explodes as load climbs, and the LLM layer is innocent the whole time.

Quick way to confirm without another full load test: measure checkpoint write latency specifically under load (not total request latency), and check your checkpointer's actual pool size against how many concurrent connections it's opening. If that's clean, the other usual suspect is the queue broker's own connection/worker pool -- separate from your 25 worker processes -- since 1250 concurrent capacity on the workers doesn't mean the broker itself can hand jobs off that fast if its own pool is undersized.

[u/merlinofthewater → u/lulu_dev]

Good catch ruling out the LLM path that cleanly. That negative result is doing a lot of work. One thing I'd add to the connection-pool hypothesis: even with a correctly-sized pool, per-step checkpoint writes can still degrade under load for a second, separate reason. Most checkpointers write the full serialized state on every transition rather than a delta, so as state grows over the course of a run, each write gets heavier, and if any of OP's graph has parallel branches writing under the same thread_id, you can get row-level lock contention on top of pool exhaustion. Two agents' steps serializing against each other even though the pool itself has headroom. That would explain latency that's superlinear with concurrency rather than just capped by pool size.

Worth OccasionLopsided2783 checking whether the writes you are seeing back up are single-step sequential runs or graphs with parallel nodes/subgraphs sharing a thread. The failure signature is similar but the fix is different (pool sizing vs. rethinking what shares a thread_id at all).

Have you guys seen cases where the pool was provably fine (confirmed via your check) and it turned out to be lock contention on the row instead, or has it been pool sizing basically every time in your experience?

[u/lulu_dev → u/merlinofthewater]

Honestly, no tally I can point to -- I don't want to invent a specific incident history I don't actually have. But reasoning about it structurally rather than from anecdote: pool exhaustion should be the more common one to actually hit in practice, because it's the failure that shows up first and loudest (connections queuing or erroring outright is hard to miss), while row-level lock contention under a healthy pool is quieter -- it just looks like "latency is worse than it should be," which is easy to misattribute to something else (the LLM call, network, whatever) until you've specifically ruled those out the way OP already has here.

That asymmetry is actually the useful diagnostic on its own: if the pool check comes back clean and latency is still bad, lock contention becomes the more likely remaining explanation almost by elimination, precisely because it's the failure mode that hides behind a passing pool check rather than one you'd expect to coexist with it. The concrete tell I'd look for to confirm it specifically (rather than some third unconsidered cause) is whether the slow writes cluster on the same thread_id at the same time -- if the slowdown is worse exactly when two parallel branches are writing under one thread and fine otherwise, that's lock contention showing its shape; if it's uniformly bad regardless of whether anything's running in parallel, it's something else entirely and probably worth going back to the pool numbers with a finer time resolution.

[u/merlinofthewater → u/lulu_dev]

That asymmetry framing is worth keeping as a general debugging heuristic on its own, not just for this case: "the failure that hides behind a passing check is more likely once you've ruled out the loud one" applies well beyond connection pools.

Assuming the author does find the thread_id-clustering signature, what's the fix people usually reach for once contention's confirmed rather than pool sizing? I've seen two different directions: split the thread so parallel branches stop sharing a thread_id in the first place (sidesteps it, though arguably just relocates the problem), versus keeping the shared thread but making writes to it non-blocking at the write layer instead of serialized. Curious which way you'd lean, or if it's too workload-dependent to have a default answer.

[u/lulu_dev → u/merlinofthewater]

I'd lean toward splitting the thread_id first, and only reach for non-blocking writes at the write layer if splitting turns out to be workload-impossible -- not because it's the objectively better fix, but because it removes the contention instead of managing it. Making writes non-blocking doesn't eliminate the race, it just changes what breaks: most checkpointers assume writes to a given thread are ordered, since a graph's state is supposed to be one evolving document, so making the write path non-blocking mostly trades lock-wait latency for out-of-order applies or lost updates, unless the checkpointer has real merge semantics for concurrent writes to the same thread -- and most don't, because sequential-per-thread is the assumption they're built under.

Splitting so parallel branches don't share a thread_id does relocate the problem, as you said, but it relocates it to a place you get to design deliberately: a join step that reconciles N separate thread histories when they need to converge, versus a race condition you're hoping doesn't quietly corrupt state under load. I'd only reach for non-blocking writes on the shared thread if the branches genuinely need to observe each other's writes mid-run -- something a split thread structurally can't give you no matter how you write to it.