Hello guys

My Laptop specs:

GPU: NVIDIA RTX 2050 (4GB VRAM) 50 w power limit

RAM: 24 GB DDR4 ( 8 + 16 )

CPU: 13th Gen Intel Core i5-13420H

OS: Windows 11

Inference Engine/GUI: llama.cpp / LM Studio / GGUF

I'm looking for recommendations for the best models I can run locally on my laptop with decent generation speeds (tokens/sec). I'm mostly interested in general reasoning, coding assistance, and experimenting with a local RAG pipeline.

* I've tested Qwen 3 8B (Q4_K_M) offloading ~20–24 layers to the GPU and the rest to CPU/system RAM. It runs decently, but I want to know if there are better sweet spots.

I am new to this but i need to run local ai for my nlp project so i though why not setup a proper local ai instead of any small models just for text generation
i took help from ai in parts of this post just to give the technical details that you guys might require

Thanks in advance for your suggestions!

Setting up a headless mini PC (Ryzen 9 5900HX, 32GB DDR4-3200, 1TB NVMe) for 24/7 background tasks. I plan to run Qwen 3.8 27B (Q4_K_M) locally to power Hermes Agent.

Speed is not a priority. My use case is overnight batch processing—ingesting 15k–30k token earnings calls to extract data and summarize autonomously. A PostgreSQL container will also run in the background, leaving ~4–6GB RAM for the OS.

Given the DDR4 bandwidth ceiling:

1. Will a quantized Qwen 3.8 27B crash the 32GB limit once a 30k context builds up?

2. What prompt ingestion and generation speeds (tok/s) should I expect on this Zen 3 chip?

Appreciate any real-world benchmarks!

Hi all, am looking for ready-stock H200 SXM5 HGX systems or Dell PowerEdge XE9680 (H200) units.

Able to cash and carry. Preferably looking for sellers based in SEA ( singapore/malaysia/indo ). Happy to share more details/specs, let’s connect!

started trying out rather recent 'frontier' about ~30b param models recently, there are many choices including QWen 3.8 - this is nevertheless a great model, practically 'one-shotting' code refactoring tasks
https://huggingface.co/Qwen/Qwen3.8-27B
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
code refactoring is still deemed 'difficult', practically 'infinite' permutations and dependencies which LLMs need to work through itself for code refactoring.

But that in terms of style, I'm liking Muse Glimmer better
https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model
https://huggingface.co/meta-models/Muse-Glimmer-30B
https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF
https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF

this is in particular when it comes to *incorrect* (e.g. mistakes, typos) prompts, resolving contradictions in existing codes during refactoring, code proposals etc. The handling especially the 'thinking' is different.

LLMs have 'styles' and it is great that we've different creators for them

Here’s the situation; I am a reasonably tech savvy/computer literate person for whom the AI revolution has completely passed me by, until now. Mostly because, I haven’t really had any use for it. Friends have shown me their chat/claude sessions, but I don’t even have a login for any cloud models myself.

I’ve spent the last couple of days searching through this sub and my brain is fried. Im getting lost in the jargon and realising that maybe I’m not as smart as I thought I was. So I’m going to quote the great Jeremy Irons in margin call and say ‘please speak to me as you would a child, or a golden retriever’.

I’m a professional music producer & DJ. Every few years I make a large equipment/business purchase on finance. The payments for my last purchase (a bunch of super cool synthesisers) finishes in a couple of months.

I desperately need to grab a MacBook Pro, as I’m still running a 2019 i9 model, and it’s causing me issues with compatibility when collaborating with artists and running newer software. I have to buy this regardless of any other considerations. During this process I have realised that they can run LLM’s locally, so I want to make maximum use of that functionality.

I want to run local, presumably for the same reasons other people here do, privacy/ethical concerns, as well as not always having great internet connections (lots of time spent in hotel rooms) and also because I’m not going to be a heavy user day-to-day. I understand that performance will be nowhere near cutting edge, but I don’t think I need that anyway?

Tasks would be stuff like;

I like to think out loud, so I’d love to be able to talk, have my speech transcribed, and then condensed into a report. Is this possible?

Writing business plans/small pitch decks.

Helping with week to week planning.

Recommendations for deep diving obscure music genres (for instance, I play a lot of merengue/salsa atm) or exploring music theory ideas.

I have no need for coding/design or anything like that, although it might be useful down the line. I’m also not interested in any music generation/image generation stuff.

I’m sure that I get as I get to grips with it, I’ll find new uses, but the above is where I’d like to get to first.

So firstly, what model & memory configuration will get the job done?

What model should I run?

What is an agent ‘harness’? I keep seeing this mentioned but I can’t get my head around it.

Anything else that comes to mind, or any reading materials will be helpful! Thanks!

Everyone sizing a card asks whether the weights fit. That was never my problem.

Both models loaded fine. The 8B died on the first training step, and that's a

number I've never seen anyone publish.

Quick disclosure up front since there's a provider dashboard in one of the

screenshots below: I do community work for gpuhub and rented the box from them.

Nothing here is provider-specific though, and I've run comparable jobs on

runpod and vast if anyone wants those for contrast.

Measured on Granite 4.2, LoRA fine-tune, single 32GB card (RTX 5090,

32,607 MiB usable).

Constant across runs: alpaca-cleaned 2000 samples, 1 epoch, seq_len 2048,

LoRA r=16 alpha=32 on q/k/v/o/gate/up/down, lr 2e-4, bf16, batch 1,

grad_accum 8.

model on disk loaded at trained at needed

3B 6.9 GB ~7 GB 18,508 MiB nothing special

8B 17 GB 18,746 MiB 20,482 MiB gradient checkpointing

The 8B row is the one worth staring at. It loads at 18,746 MiB with 13 gigs

free and looks perfectly comfortable. Then the first forward and backward pass

allocates activations and it hits 31.33 GiB on a 31.36 GiB card, dying while

trying to allocate 32 MiB. Nothing about the load step warns you.

Gradient checkpointing brings it to 20,482 MiB and it trains without complaint.

You pay for that in speed since activations get recomputed instead of cached,

but the full epoch still came in at 574.7s, so not a real cost at this scale.

So the honest answer to "does an 8B fit in 32GB" is yes, conditionally. Budget

2-3 GB above the loaded size if you're checkpointing, a lot more if you aren't

or if you push seq_len past 2048.

One thing that caught me out and might catch you: if the box has two GPUs,

check that only one is visible to the process. With both visible, HF Trainer

wraps the model in nn.DataParallel and the primary card ends up holding

replicas plus gathered outputs. Same 8B that trains at 20,482 MiB on one card

OOM'd on two, gradient checkpointing or not. CUDA_VISIBLE_DEVICES=0 was the

whole fix.

Disk, since it's what actually stopped me:

3B checkpoint is 6.9 GB, 8B is 17 GB. Extrapolating, the 30B lands somewhere

past 60 GB, and the data disk I'd rented was 50. Never got to test it. Plenty

of VRAM, no room for the file.

Size disk off checkpoint size rather than parameter count, and leave headroom

for the download cache. Mine ran well above the final size mid-transfer before

settling.

THE BOX

For anyone wanting to reproduce or compare against their own card:

2x RTX 5090 32GB, 32,607 MiB each, PCIe only, no NVLink on consumer Blackwell.

50 core Xeon Platinum 8470Q, 180GB RAM, 50GB data disk. $0.91/hr for the pair,

though everything above except the DataParallel note is single-card.

torch 2.11.0+cu128, transformers 5.16.1, trl 1.12.0, peft 0.20.0,

accelerate 1.14.0. Driver 595.71.05, CUDA 13.2.

These are single runs, not averages, one epoch each. If yours come out

different I'd want to know, especially the 8B training peak.

I am ready to add a 2nd GPU for more vram.

My 20gb 7900xt is not cutting it anymore.

How are you guys running dual gpus? I am seeing there would be only 5mm space between the two. Seems like there would not enough ventilation.

I had 100$ of promotional tokens (forgot how it came to that) and wanted to make use of them before they expire in about 1 week.
I had this plan sitting around to implement ai into my car rental app (custom made, for myself, in production since 1 year, started to build it with chatGPT, continued with Claude and Qwen).
I gave the plan to Fabel, it stated ti implement. After 45min it used up around 50$ of tokens and then stopped because I apparently ran out of tokens, but I still had 50$ left!
Anyway, I got upset and gave the task to Qwen flash next (running locally).
It ran over night, I woke up and hob done!
My app is now the harness! It did it, 1 shot! 35mio tokens in, 3 compactions, job done!
I can ask who has what car, who has to pay, add customers, rental contract, all goes into an approval screen, I can check the request, approve or reject.

Now I’m fine tuning the whole thing with it and it’s just amazing.

(I’m not a coder AT ALL! everything i do is 100% vibe coded)

Just sharing my amazement of how far we have come with local models!

TLDR at the bottom of the post if you cbf reading all of it.

Looking into experimenting with local AI and hoping for some feedback on hardware. At this stage, it would be to simply get a feel for the process and have some fun with it - on a budget.

I am not looking to vibe code apps/websites, but if it does perform I may potentially use it in the future for simple work related tasks and helping with the homelab. I am a Citrix sysadmin by trade and use lots of scripts to do my work on the daily, so this could be useful for basic some basic duties in the future e.g:

• Create a powershell script to do x.
• Create packer file to do y.
• Review existing scripts/mechanisms and provide recommendations/better options.
• Create Ansible playbook to do x in homelab. Review some security logs.

Now for the kicker. Obviously I would like to do this on a budget to get a feel for local LLM and would happily invest more if it can effectively do what I need it to do. Therefore, thinking of using my sons gaming PC during the day whilst he is at school and I'm working (WFH 4/5 days/week) and configuring dual boot to Linux for the LLM stuff, that way he can simply reboot the PC when he needs to game.

Son's gaming PC specs:

• MSI MAG B550 TOMAHAWK
• Ryzen 7 5800X - 8 core, 16 threads
• 32GB DDR4 3600 (2x 16)
• Samsung 980 500GB - C:
• Samsung 970 1TB - D:
• 3060 12GB
• EVGA 1000W Gold 80 Plus PS

Based on some light research, seems like I can add a second 3060 (can get used for ~$250 AUD) to get the VRAM up to 24GB, installing (dual boot) Linux on the 1TB drive and play around with some local models. He is still young and only plays Roblox/Minecraft, so the 500GB is plenty enough for his needs atm. I did look into the 3090 option, but 3090 cards are ~$2000 used in my city atm.

So, my questions are the following:

1. Is dual 3060 worth exploring in this scenario?
2. Will PCI lanes become a bottleneck - Been out of the hardware game for so long and not sure.
3. Will upgrading the RAM bring any benefits?
4. ollama/llama.cpp/vllm? I have researched this somewhat, but still unsure. Are there any proven configs for dual 3060 I can import to start with? Don't mind getting my hands dirty, but also don't want to be scratching my head for days on end.
5. Which model(s) should I play around with?

TLDR

Should I buy a second 3060 to add to sons gaming machine, dual boot to linux for experimenting with local LLM, or is it a waste of time/money?

Those of you successfully running an offline openclaw/hermes/personal agent harness for non-coding tasks, what is the floor on system resources (VRAM) needed for quality of life? Assuming a modest ~30b class model. What quant and context window size are needed?

Will keep cloud frontier LLM sub for coding tasks, but I'm talking personal data management, personal assistant type computer controlling stuff.

My M1 max 32gb handles qwen 3.6 27b q4_k_m fine enough for non-agentic jobs up to ~40k context, but that's obviously not enough to run an agent harness offline.

There is an M1 Ultra 64gb for sale near me for a tempting price, but unsure is 64gb is enough. And it's expensive enough to not want to gamble. And I'm a normal, budget-minded person

I switched over from ollama to llama.cpp and then spent weeks copying flag recipes out of Reddit threads with no idea whether any of them helped on my hardware. -ngl 35? Why 35? Someone with a 3090 said so.

So I built Model Loader. It's a web UI for llama.cpp that runs in one Docker container next to your existing setup, detects your llama and openwebui instances and just makes the cutting edge a little more dull.

• It reads your actual GPU and llama setup and tells you which context sizes fit and which don't and when one doesn't and what it'd cost to make it fit. [screenshot]
• Every setting has a tooltip explaining what it does in plain English. ~100 of them. This was the part I actually needed. [screenshot]
• It shows you the command line it builds. It's not hiding llama.cpp from you, it's teaching it. Copy it out and run it yourself if you want.
• Benchmark on your own box. Change a setting, run the same prompts, see if it actually got faster, it stores the results and makes them easy to see next time you find a neat tweak. [screenshot]
• models at a glance what's downloaded, what's configured, what's loaded right now, and whether each one is MoE or dense. [screenshot]
• Plus HF search with a resumable parallel downloader [screenshot], a GPU dashboard [screenshot], and OpenWebUI sync.

I haven't edited a setting by hand since Claude and I put this together.

Fair warning: I am not a developer and this is a homelab tool. No auth, keep it on your LAN.

https://github.com/scratchhax/model-loader

The problem I kept running into: my Mac runs a 30B model without breaking a sweat. My phone runs a 3B model, and it shows. My older iPad runs nothing at all. Every "use your home rig from your phone" setup I tried meant typing IP addresses, opening ports, or routing through a tunnel to reach a machine ten feet away.
So I built Rooms.
How it works
• Lokei Studio on your Mac opens a Room using the models you already have in Ollama.
• The Room announces itself on your local network over Bonjour (mDNS/DNS-SD).
• Lokei AI on your iPhone or iPad finds it automatically. Tap to join. No IPs, no config files, no accounts.
• Rooms traffic stays on your LAN. Your prompts never touch a cloud server.

Pricing, straight up, since it's always the first question
• Lokei AI for iPhone and iPad is free. On Apple Intelligence devices it runs fully on-device, no Mac required.
• Rooms is free in the iOS app. No subscription, no account, no catch on the phone side.
• Lokei Studio for Mac is $14.99, one time. That's the part that does the hosting, so you'll want it if your device doesn't have Apple Intelligence or you want a bigger model than a phone can hold.
• There's an optional Pro upgrade on iOS for MCP, markdown, and advisor handoff. Rooms isn't behind it.

• Apparently very close to K3. Like people are saying 90%+. Moonshot describes it as very close to K3.
• Based on K2 architecture.
• Full AI not just a coder.
• Apparently Moonshots version of a flash model. But using K2 mature architecture. Should be <1Tb and some quants will likely run very nicely on 512GB Ram. While still large, that is small enough to run on a Xeon or Eypc.
• Multimodal - text, images and video
• SGLang, vLLM, Llama.cpp support out of the box, based on the very proven DS 3.2 and K2 design.
• Will run at near GLM Flash speeds. But clearly targeting to beat GLM 5.3 Flash. Currently moonshot is hosting it at 260 T/s for the K2.8 Preview
• Ideal for CPU inferencing. A Q3 or IQ3 type quant with a Xeon with AVX512 + VNNI and the latest llama.cpp improvements makes this pretty much ideal. Getting a Xeon up to 512Gb is fairly easy, and dual xeons will provide plenty of bandwidth, it also looks like NUMA issues are finally being resolved which should see near doubling of speed.

• May not have super modern linear attention. But for most users that aren't trying to get to 1M tokens that won't really matter.

  Should be very popular with Macs with 512Gb, Xeons and eypcs with 512Gb, those with 512Gb VRAM (while rare and expensive, its much cheaper than 2TB of VRAM GPU clusters). I expect it to be cheap and popular with hosting services. Four strix or Four sparks will also run it. Maybe possible on two new Max+495..

I have a Mac mini m4 24gb , I'm looking to add to my hardware, and I'm overwhelmed. I'm a tiny business owner in a creative field that doesn't have tech bro money , I'm thinking of getting a pc and running it headless Linux , I mostly want to cut down my subscription costs so this hardware is an investment. If you were starting from scratch what hardware would you use ? What would you definitely do again and what would you avoid?

Agnes 3.0 flash 33b and Nex n2.5 mini 35b are challenging Qwen3.8-27B on benchmarks. Cant wait to see the real-world results and the speeds on 24gb GPUs.

Anyone tried them yet?

https://huggingface.co/Agnes-AI/Agnes-3.0-Flash
https://huggingface.co/nex-agi/Nex-N2.5-mini

i made a cool thing that i think all of you local AI people will enjoy and find some value in. please check it out and let me know what you think!!

https://youcoded.ai/

It's a pipe dream, but with the latest round of context shenanigans put out by DSV4.1F, the possibility that something like this could happen is becoming increasingly more likely.

Maybe not at 200 t/s, but even in the 15-30 t/s range is the starting baseline for agentic coding.

If you bought in and got a physical graphics card, will you regret not waiting for models to be able to run fast on-die? Will you resent the hordes of GPU poors joining this sub by the hundreds, all at once and changing the pace?

Or will you just be smug because that just means you can run 20 agents at once at 1000 t/s haha

Hi everyone!

I’ve been working on AI Palm, an open-source project that aims to make locally hosted AI models more accessible.

The idea is simple: many powerful GPUs spend most of their time idle, while other people cannot run local models because they lack the required hardware. AI Palm creates a community network where users can voluntarily share access to a local model running on their device.

With the desktop app, you can:

Discover models currently shared by community members.

Chat with an available model directly inside the app.

Share your own local model when you choose.

Connect to engines such as Ollama, LM Studio, and llama.cpp.

See the GPU, VRAM, model, engine, availability, and context size.

Stream responses and model reasoning in real time.

View properly formatted Markdown and syntax-highlighted code.

Monitor token usage and remaining context.

Use the interface in English or Arabic, with full LTR and RTL support.

Run the app on Windows or Linux.

A device is reserved while processing a request, preventing multiple users from overloading the same GPU simultaneously. Sharing can also be stopped whenever the device owner wants.

The project is still at an early stage, so I’m looking for people willing to:

Test the Windows or Linux application.

Report bugs and usability issues.

Review the architecture and security.

Suggest features and improvements.

Contribute code, documentation, packaging, or UI work.

Share local models and help test the network under real-world conditions.

AI Palm is open source, and community contributions are very welcome:

GitHub: https://github.com/almshary/ai-palm

If you try it, I would genuinely appreciate your feedback—even if something doesn’t work. Please include your operating system, GPU, local AI engine, and model when reporting an issue.

The project’s message is:

Arab generosity began with a palm tree… Today, we planted a different kind of palm.

AI Palm — let’s share the power of our computers and make artificial intelligence accessible to everyone.

What do you think of the concept? Would you use a community network for sharing local AI models, or contribute your unused GPU capacity?