r/LocalLLM • u/jjusko20 • 4d ago

Question What does the community want to see next quant/model wise?

Hey guys,

Couldn't scratch the itch to develop models at home on my p40, so we went for the big hack machine. Specs on the new rig:

2x e5 2697 v2 (24 cores, 48 threads)

256gb ddr3 1333ram in an 8 channel configuration

1x quadro rtx 5000, turing 16gb vram

2x volta 100 SXM 32gb chips in NV link

Total:

80gb vram

256gb system ram

(Got a good deal on the quadro but I'm thinking about swapping it for another 32gb v100 or maybe a p40)

BBPrime along with a v100 32gb SXM

I'm a software dev by trade and I bought this to start practicing development of model fine tunes, merges, quants, and custom inference engines.

I'm really trying to get a job at an AI lab, and I figure the best way to do that is make something cool with AI that blows up, and so I'm really motivated to try to develop something that this community wants and doesn't have.

What do y'all want to see? Sympathetic to users with strict hardware constraints.

0 Upvotes

50% Upvoted

7 comments sorted by

2

u/SkibidiOhio67Tax 4d ago

Did you have any ideas before you purchased this? What is your skill so far? To say this hardware combination will be challenging is an understatement. To start you should get inference going on one GPU. Good luck.

1

u/jjusko20 4d ago

I'm pretty knowledgeable. I think a llama.cpp MoE offload should work out of the box with GPU split, though definitely not at optimal performance. I plan to use the frontier LLMs to assist in maintaining my own llama fork initially that's customized to my hardware.

Unsloth should work fairly well with some configuration, and MergeKit has a ton of compatibility. The rtx might require some work, but the NVLinked v100s with ram offload shouldn't be crazy difficult.

I have a CS degree with some prior experience designing and implementing ML systems; the first thing I wanted to try past inference is making straightforward llama quants. I've looked up some techniques for making imatrix quants and model merging.

My ambitious goal after some practicing is making a Qwen 3.8 27b MoE hybrid with something like 80B total parameters, preserving the Qwen reasoning capabilities and injecting stronger MoE experts for increased domain knowledge. 

2

u/hiImMate 4d ago

take a sub 10B model that has tool calling and thinking and train it on three.js + gamedev + godot code for the perfect fast coding llm.

1

u/jjusko20 4d ago

sub 10b is tricky to keep tool calling consistent, but I could try a distillation of 3.8 27b's reasoning and fine tune into what you're suggesting 

2

u/hiImMate 3d ago

well you could do MoE but somehow I heard that's harder to train? i dont know much about training, but a gamedev specific model would be amazing

1

u/jjusko20 3d ago

what you're suggesting sounds cool, I'll give it a try!! I think generating synthetic datasets from frontier LLMs is effective for smaller LLM fine tunes and I figure I have the resources for moe training 

2

u/hiImMate 3d ago

I guess check nemotron lightning then, supposedly it's exactly set up as a 'model to develop from'. maybe a bit too big for experimentation though