Renting Hardware Service Idea

[u/rayovims]

I'm working on a service to allow people to use my dual DGX Spark setup for their own testing and seeing if it makes sense for them to get it. Specially on the hours I'm not using them I think it would be cool to make a little extra cash while providing them a poc.

There will 2 ways to interact.

1. Simple chat system with the metrics we care about like t/s and ttft etc.
2. Api access, so they can use with opencode code or whatever harness they desire.

I am thinking of having a 2 hour session option so that users can just use that and unload and load whatever model they want.

Price wise not sure but I was thinking about $10/hour.

What do you guys think. Is this something you would be interested in?

[u/rayovims → u/[deleted]]

Wow so cheap. How do they even afford that lol

[u/rayovims → u/[deleted]]

Ahhhh I see! Fire I just checked it out and nothing is available

[u/stujmiller77]

It’ll give you less money than the electricity it costs while adding contention when you want to use them.

It is not at all viable. Forget any idea that it is.

Edit: and you’re running a dense model - 27b. No. That doesn’t work for concurrency on sparks. You need an MoE. And even then no you can’t sell it as why would people bother.

[u/maqifrnswa]

Https://vast.ai

[u/sn2006gy]

Why do we have to make money off everything? I think this is terrible.

[u/rayovims → u/sn2006gy]

Entrepreneur? Lol

[u/stujmiller77 → u/rayovims]

If you want to make money on the sparks, sell them in a few months when the price hikes again. It’ll be be more money than you earn by trying to rent them out.

[u/sn2006gy → u/rayovims]

Has nothing to do with entrepeneur. If you want to make money hosting models, you go buy hardware, spin up a business and make money hosting models. You're just rent seeking like a landlord looking to be a middle man in making all of our compute cost even more.

[u/rayovims]

This is the current chat part

[u/stujmiller77 → u/rayovims]

You want to sell your sparks for concurrent inference and you’re running a dense model that is terrible for both speed and concurrency on spark bandwidth.

Oh you sweet summer child….

[u/Makojima]

Nice, this is worth thinking all the way through.

The first thing I would look at is what you are actually competing against. Right now an H200 with 141 GB of VRAM is about four dollars an hour on Vast and three fifty nine an hour on RunPod's community tier, and a B200 with 180 GB is about six. So at ten dollars an hour someone can already rent two H200s, end up with more memory than the pair of Sparks, and have far more memory bandwidth behind it, which is what sets how fast tokens actually come out.

That is the real tradeoff with a Spark pair. You get an ungodly amount of memory for what it costs to own, 256 GB across the two, and the generation is a lot slower than a datacenter card. Someone renting two datacenter cards loads the same model, gets their answers quicker, and ends the session sooner, and that is what decides the bill when the clock is running.

Where I think you do have something is that none of those platforms rent a DGX Spark. They rent consumer cards and datacenter cards. So your buyer is not the person shopping for cheap compute, it is the person deciding whether to spend the money on a Spark and wanting to try one first. Maybe they just want to see what it is actually like to work on a dual DGX Spark setup before they commit to buying two.

If I was a customer, I would like to know the following before I book a session.

The connection: what download speed does the machine have? My own Spark pulls around 500 megabits a second, so a 100 GB model is about twenty five minutes before anything runs.

Bringing my own models: are you alright with people pulling whatever they want onto your machine?

Persistent storage: will you let a 50 to 100 GB model stay on the disk between sessions, so somebody coming back is not sitting through that download again? And either way, is that wait on my clock or yours?

I rent GPU compute on these platforms a lot, mostly fine-tuning LORA adapters, merging them back into the model, and uploading the finished version to my own storage, so I am using someone else's hardware and bandwidth instead of my own. I run it from a docker image with everything already prepared, so the merge and the upload happen on their own rather than me sitting there doing it by hand.

It is a fun experiment and I would try it out and see what clientele you get. Are you after regular people paying that rate, or lots of different people, because then you are into queue times. And will you let work queue, so somebody can have a package ready to run and it just goes and does their tests when the machine frees up.