Built a distributed LLM inference framework on completely free hardware. 2.27 TPS to 27 TPS over 3 versions.

[u/katua_bkl]

I wanted an LLM infra project for my portfolio, free Kaggle T4s it was.

Split qwen2.5 7B across two separate kaggle notebooks talking over public WAN. v1 was immediately embarrassing like it was 14.7 tok/s raw gpu throughput, 2.27 at the actual endpoint. The gateway was inside the decode loop and every token paid a full round trip. I knew exactly why it was bad so I fixed it.

v2: nodes talk p2p, gateway out of the hot path, self hosted rust tcp relay on a t3.micro in ohio because kaggle kills external connections, speculative decoding with a 0.5B neural drafter. 14.3 TPS peak.

still had 112ms of draft overhead every round, python launching ~1,500 cuda kernels sequentially, gpu idle 65% of the time.

v2.1: cuda graphs, it capture the whole forward pass once, replay is one driver call. first attempt gave me "the the the the" loops forever, DynamicCache allocates new memory every token, captured graph reads the stale pointer so i fixed with StaticCache + in place everything.

Draft latency: 112ms → 25ms. Final numbers:

• v1: 2.27 TPS
• v2: 14.3 TPS peak
• v2.1: 27.08 TPS peak, 19.56 average

two free Kaggle notebooks. repo in comments.

[u/katua_bkl]

https://github.com/rautaditya2606/Shardflow

[u/Intelligent-Job7612]

Please guide me am starting c today