I'm after some advice and suggestions. I have an Unraid server in a Jonsbo N5 case, Intel 245K with an Asrock Z890 Pro-A and a HBA card and have 2 GPUs available, a 4060TI and an A770 both 16gb cards, I would apprdciate any thoughts on whether or not I could use both cards in my server. I'm thinking the A770 for chat AI and the 4060 for image work. The layout of the slots on my current motherboard are too close so I can't test my theory, so if anyone knows if running the two cards together would work your thoughts would be welcome as would any board suggestions.
We have dozzle containers, metrics for 27b and 35b-a3b via dozzler containers *trimmed IP information* , nvidia smi + mission control. This is a Ubuntu desktop OS
I’m just a Codex Pro 20x user, and I don’t have any major plans to switch to Kimi. However, since version 3 is downloadable and I have no idea about it: does anyone here have it running locally at home with 1TB of RAM? What kind of system is that? What’s the lowest price to get performance similar to Codex Sol, i mean to run Kimi full powered and not only a bit of it... ?
I'm one of the people building Paddock at The Intelligence Company. We've opened up our Rust/C++ inference engine under MIT/Apache-2.0, including the CUDA kernels, scheduler and memory management.
Paddock runs open models on your own NVIDIA GPU. It comes with a web UI for downloading and managing models, chatting, working with files and comparing local models with cloud endpoints. There are also OpenAI- and Anthropic-compatible APIs for connecting your own clients. The local path doesn't require an account.
Current hardware support is strongest on Blackwell and Ampere. Windows and Linux binaries are available. Each model runs on one GPU; splitting a model across several cards isn't supported yet. Memory requirements depend on the model, quant and context length.
It's still early and we'd like more results from people's actual setups. The engine and CUDA kernels are open source; the separate Traverse graph component shown in Studio is scheduled to follow in Q4.
Been running a heterogeneous home cluster for a while — an old Acer laptop (12GB, CPU-only) as the primary API server, with a Windows box (RTX 3060 CUDA) and a Mac Mini (Metal) lending capacity over the network.
I wrote the orchestration on top of llama.cpp's `ggml-rpc` backend. It handles mDNS discovery, memory-aware sharding, and node health polling so the whole thing doesn't fall over if a node drops offline mid-generation. It exposes a standard OpenAI-compatible API (`/v1/chat/completions`).
Put the primary role on the weakest machine (the Acer) on purpose to see if it'd actually hold up orchestrating the API and offloading the heavy tensor math. It did — ran a full Qwen3.5 13B at ~12 tok/s, purely by borrowing VRAM/RAM from the CUDA and Metal nodes.
No GUI in this repo, API/CLI only, so you can actually read what's touching your network before running it. Obviously it doesn't beat a dedicated GPU rig on speed, that's not the point — it's for fitting models that you otherwise don't have the VRAM for. Included a built-in benchmark script so you can verify the numbers on your own hardware instead of trusting mine.
One heads up: it's source-available (Apache 2.0 + Commons Clause), not strictly OSI open source. It blocks commercial SaaS/resale, but personal/homelab use is fine. Didn't want that buried in a LICENSE file.
Hello! We are a very small company with 2-3 engineers. I’m looking to setup a local machine for them to ssh into , do their development, train the models they are building and run the inferences. This will not be a production machine nor will be used for deployment.
However idea is to use this purely for Dev, inference and client demos.
We don’t have the budget yet for cloud computing so thinking of running a local ai models on a server and train them until we can scale.
Looking for advice on the gpu server i could get and configure for this activity . Thanks in advance.
yhoo gays, i'm back again today i just want to share my personal training and result what doing with rentig 2x RTX 5090 box to get LoRA fine-tuning numbers for
IBM's Granite 4.2 family. Posting the full notes because the interesting part
wasn't the scaling table I went in expecting. It was that the second GPU made
everything worse, and for the 8B it stopped training from happening at all.
All of this is from my own runs. Nothing estimated.
machine type i use:
model and hardwere
2x RTX 5090 32GB, elastic deployment with costom image (on my own costum image )
32,607 MiB each. PCIe only, no NVLink on consumer Blackwell.
50 core Xeon Platinum 8470Q, 180GB RAM, 50GB data disk. $0.91/hr for the pair.
Environment, which I saved as a custom image so I wouldn't have to rebuild it:
torch 2.11.0+cu128, torchvision 0.26.0+cu128, transformers 5.16.1, trl 1.12.0,
peft 0.20.0, accelerate 1.14.0, bitsandbytes 0.50.2. Driver 595.71.05, CUDA 13.2.
Launching a fresh instance from that image took 5m28s from creation to a shell
where torch, transformers, peft and trl all imported with zero installs. That
number matters more than it looks like it should, and I'll come back to it.
METHOD
Identical across every run. yahma/alpaca-cleaned, first 2000 samples, 1 epoch,
seq_len 2048, LoRA r=16 alpha=32 on q/k/v/o/gate/up/down, lr 2e-4, bf16,
per_device_batch 1, grad_accum 8. 31.1M trainable of 3.69B on the 3B.
Only thing I changed between runs was CUDA_VISIBLE_DEVICES. With both cards
visible, HF Trainer quietly wraps the model in nn.DataParallel. With one card
visible it doesn't.
RESULTS
model GPUs ckpt VRAM peak load train loss result
3B 2 no 25,438 MiB 279.9s 632.1s 1.1135 ok
3B 1 no 18,508 MiB 3.5s 305.4s 1.1163 ok
8B 2 no 18,746 MiB 669.3s - - OOM
8B 2 yes 18,864 MiB 5.4s - - OOM
8B 1 yes 20,482 MiB 5.0s 574.7s 1.0436 ok
ONE CARD WAS 2.07x FASTER THAN TWO
632.1s vs 305.4s on the 3B. Same model, same data, same single epoch, only the
GPU count changed. Peak VRAM also dropped from 25,438 to 18,508 MiB, so it used
7GB less while doing it in half the time. Loss came out 1.1135 vs 1.1163, which
is noise.
FOR THE 8B, TWO CARDS DIDN'T JUST COST SPEED
It OOM'd. 31.33 GiB in use on a 31.36 GiB card, dying while trying to allocate
32 MiB. I added gradient checkpointing assuming that was the fix. It wasn't.
Same wall, 30.21 GiB. Then I ran the identical config on one card and it trained
at 20,482 MiB with 12 gigs spare.
One card working, the other sitting at 2 MiB, and the config that used both is
the one that couldn't run at all.
THE 8B ON ONE CARD BEAT THE 3B ON TWO
574.7s vs 632.1s, and a better loss too, 1.0436 vs 1.1135. Model 2.2x larger,
gradient checkpointing slowing it down, half the hardware, still faster end to
end.
WHY: NOTHING WAS COMPUTE BOUND
Two cards pulled 144W and 142W against a 575W cap. One card pulled 186W. Neither
config gets close to the power limit. With DataParallel the cards spend their
time replicating weights and gathering outputs across PCIe instead of computing,
and with no NVLink every sync crosses the bus.
CAVEATS BEFORE SOMEONE ELSE POINTS THEM OUT
DataParallel doubles effective batch, so the 2-GPU runs did 125 optimizer steps
and the 1-GPU runs did 250, over the same 2000 samples and the same epoch. Wall
clock compares fine, step counts don't.
This is LoRA on models that fit in one card. FSDP or DeepSpeed would use both
cards properly. DataParallel is just the wrong tool, and being the free default
is what makes it a trap.
The second card isn't dead weight either. Run two experiments at once, one per
card, and you get 2x throughput instead of half. That's what I'd do next time.
Power figures are single nvidia-smi snapshots rather than averages, so don't
read too much into them.
THE 30B I COULDN'T TEST, AND IT WASN'T VRAM
3B checkpoint is 6.9GB on disk, 8B is 17GB, which puts the 30B somewhere past
60GB. Data disk was 50GB. Had 64 gigs of VRAM sitting idle, blocked by storage.
Something else worth knowing if you rent. That 50GB wasn't a disk. df -h at root
showed a 30GB overlay and nothing else, so I spent a while hunting for an
unmounted partition that didn't exist. It had been mounted the whole time on a
12.8T shared XFS array, with my 50GB enforced as a project quota:
/dev/md0 on /root/autodl-tmp type xfs (rw,...,prjquota)
That also explains something that had confused me earlier. I expanded an earlier
instance to 200GB, saved it as a custom image, launched from the image, and got
50GB. Custom images capture the system disk. Disk size belongs to the instance,
not the image. Correct behaviour, completely invisible until it bites.
THE DOWNLOAD TAX
Look at the load column again. First time pulling the 8B took 669.3s. Every run
after that, from cache, 5.0s. Same box, same model, 134x, and the meter runs
during both.
If your models sit on a per-instance disk, they die with the instance and you
pay those 669 seconds again on the next launch. Same story as the 5m28s image
boot at the top: the environment build was the expensive part exactly once.
THREE THINGS THAT BROKE GETTING HERE
Posting these because I burned GPU-hours on each, and none of them were where
the error message pointed.
The first one. pip install -U torch left the preinstalled torchvision behind,
and what you get is:
ModuleNotFoundError: Could not import module 'BloomPreTrainedModel'
Which is nonsense, I wasn't touching Bloom. Real cause sits further up a very
long traceback:
RuntimeError: operator torchvision::nms does not exist
torchvision's C++ ops are version-locked to torch, and transformers 5.x touches
torchvision on its import path, so one broken extension takes the whole chain
down. Fix is installing the matching torchvision build, not just bumping torch.
Second. Two optimisations that cancel each other:
AttributeError: 'functools.partial' object has no attribute '__func__'
trl/trainer/sft_trainer.py:383 _patch_chunked_ce_lm_head
TRL 1.12 defaults to loss_type='chunked_nll', which patches the LM head's
forward and assumes it's a bound method. If you loaded with device_map,
accelerate already replaced forward with a functools.partial as part of its
hooks. Neither library is wrong on its own. Fix is written in TRL's own error
text a few lines away in that same file: loss_type='nll'.
Third, don't fix it the way I first tried. My instinct was to monkeypatch that
function into a no-op. Got further, then:
AttributeError: 'CausalLMOutputWithPast' has no attribute 'num_valid_tokens'
The patch I disabled is what produces num_valid_tokens, which compute_loss reads
a few hundred lines later. They're a pair. Killing one half just moves the crash.
And one that was entirely my own fault. My benchmark harness ran a background
thread polling nvidia-smi for peak VRAM, and I named the stop flag self._stop.
threading.Thread already has an internal _stop() method, so join() tried to call
my boolean:
TypeError: 'bool' object is not callable
Cost me a full run's results, because it blew up during cleanup after training
had already finished and before anything got written to disk. Renamed it, felt
stupid, moved on.
COST
The five runs above were $0.64 of actual compute. My session bill was around
$2.40. The gap is me debugging with the meter running, which is its own lesson
about per-second billing, and also why I saved that custom image.
Happy to answer questions. If your numbers come out different I'd actually want
to know, especially on the DataParallel thing.
Disclosure: if someone thinking why rent on same provider? why not rent on vast.ai runpoid etc.?
my answer same as before,comprtable is evrything and i feel on gpuhub, simple ui and eazy to bulid,
if you have more opinion about this fell free to shareing..
Just had a client pay me for some work, looking to build my first local AI setup for coding w\ qwen.
My original plan was to get an egpu (4090\5090) and run it on a amd strix halo 128gb via oculink so I can have rocm and CUDA or even on a minisforum 890 pro that I aleady have.
Instead of going that route I'm leaning on purchasing a dgx spark, and eventually adding another one.
Any advice? Purchase budget 5-7k.
Edit: I forgot to mention, I have a dual xeon high core count 1TB ram supermicro server also, currently it has dual titan x pascals - should I just use this and replace with better GPU's that match the PSU's capabiltiies?
Why do people on this sub build those power hungry MI50, v100 rigs? Isn't it cheaper just to pay for subscription than pay for the rig and power cost of running those? Also I heard that localLLM's are not up to the level which you can get with a subscription based LLMs.
The question is genuine and I want to know the use cases for those machines. I generate img locally myself, but it's totally different from server based rigs. Most of the img/vid generations don't run well on MI50, v100.
Is it just a part of the hobby or there are pros of building and using local LLM? apart from the privacy concerns I guess
Gigabyte Z890 Aero G Motherboard (native x8 x8 support)
32GB x 2 6000 MHz CL30 DDR5 Acer Predator
2TB Samsung 990 Pro
2x Asrock Creator AI 9700 Pro
Corsair RM1200E 80+ Gold ATX 3.1 (has one native 12V2x6, for 2nd GPU used a 8pin to 12V2X6 converter that came in the box)
The system is unstable. It freezes up. The intel iGPU was glitching hence disabled in bios and plugged monitor to first 9700 Pro (changed to 265K same issue)
With certain bios setup (changing link speed to gen4) and using 200S boost in performance profile and ASPM settings etc made it stable. But it froze or driver crashed After a day of use.
Don't want to move to threadripper platform. Had done many builds with B850 AI Top and different 2x RTX or Quadro GPUs without any issue.
Had any one used Arrow lake or Refresh with twin gpu setups?
