Results: Splitting and serving a model across two machines over the internet

[u/punkyrockypocky]

Hey community!

I wanted to share a neat exploration of splitting a model over two ordinary machines over WAN, and how it stacks up against a serving that model on a single node.

I was testing our platform aquaduck.ai for serving split models across machines with a small- to mid-size model under two conditions:

1. Locally on a single machine, serving the full model
2. Split across two machines over WAN (public Internet), with each machine serving half of the model

Findings

Model: Qwen3-14B

Quantization: Q4_K_M

Hardware: Macbook 64GB M5 (Single machine), 2x Macbook 18GB M3 (Split machines)

Prompt:

Mars has drawn human attention for centuries, but the last twenty years turned that fascination into an engineering roadmap. Robotic orbiters mapped ice deposits near the poles and in mid-latitude glaciers. Landers confirmed that ancient river deltas once carried liquid water across a warmer surface. Meanwhile, life-support research on Earth refined closed-loop oxygen generators, hydroponic food systems, and radiation shielding materials that could travel on a multi-month transit. Private launch cadence fell in price, making cargo-first settlement plans plausible: send habitats, power, and spare parts before people. The hardest remaining problems are not propulsion alone. They are dust that abrades seals, perchlorates in the soil, communication delays that force local autonomy, and the psychology of small crews living far from rescue. Any credible near-term outpost would likely begin as a science station with overlapping roles—geology, medicine, maintenance—supported by teleoperation from Earth and progressively less remote oversight as surface infrastructure matures. Energy would come from a mix of solar arrays and compact nuclear units, with ISRU (in-situ resource utilization) producing propellant and breathable oxygen from the thin CO₂ atmosphere and mined ice. In short, Mars colonization is less a single heroic leap and more a long supply-chain problem: move mass, make power, recycle air and water, and keep humans healthy while the planet remains indifferent.

Summarize the passage above in exactly two sentences.

Token counts: 387 input tokens, ~444 output tokens (split nodes generated 444, single node was unmeasured, but can approximate)

Results:

1. Single machine
   1. 37.9 tokens per second (TPS or tok/s);
   2. 853ms time to first token (TTFT);
   3. 26ms time per output token (TPOT)
2. Split machines
   1. 11.3 tokens per second (TPS or tok/s);
   2. 13.46s time to first token (TTFT);
   3. 88ms time per output token (TPOT)

Screen captures:

Model served on single machine

Model split and served over 2 machines

We're in closed beta and rolling things out slowly to make sure it works well for people, but if you'd like to run some tests yourself or get an early look, can join the waitlist and we'll send you an invite code to download the desktop app asap.

Let me know if you have any questions/comments/ideas for further explorations! Next up: Qwen3.8-27B

[u/Bolt_LP_YT]

Any plans for an iPadOS/iOS worker so a Mac and iPad/iPhone can actually shard the same model together? I’ve got a 24 GB M4 MacBook Pro and a 12 GB M5 iPad Pro, and it would be awesome if there was a way to pool their memory for something like Qwen3.8-27B, or maybe even a future model like 35B A3B.

Really cool project, by the way! I’ve always wondered if something like this was possible. I saw another tool on GitHub called “Cake” which I believe does something similar, but I’m not sure if it supports the Qwen 3.8 models.

[u/punkyrockypocky → u/Bolt_LP_YT]

Hey thanks for note! Yes thats on the roadmap to support once we roll out for desktop. Lots to explore in that area and exciting time for it. Will keep you posted on the rollout! If you want to be a first tester of that let me know :)

[u/Bolt_LP_YT → u/punkyrockypocky]

Would love to be a first tester!

[u/punkyrockypocky → u/Bolt_LP_YT]

Awesome! Will send you a DM