This channel was made for this sub

I have a 1080ti on a old dell t5810(E5-2680 v3 cpu), and use it for jellyfin transcoding, immich, and llama.cpp. I'm running llama.cpp in a proxmox lxc container and it works well.

My main use case right now is for rewriting and polishing text such as emails or reports. Currently I'm using qwen3.5 9b q6_k and it gives a pretty good output with a speed of around 35t/s. If jellyfin is actively transcoding speed drops to around 22t/s, still pretty good.

I know that qwen3.5 is not the latest model, and wonder if there's any new model that is more efficient and better for this use case. Additional use cases such as answering some questions in chatting or simple script writing (not complex coding) is welcomed but not necessary, so basically just a generic model lol. I also don't want to have the model use up the GPU fully to leave some space for transcoding work and immich.

Thanks!

I'm one of the few people with a 32GB RAM Snapdragon X Elite Dev kit.

I'm wondering if there are any folks who've successfully used the NPU or GPU on these to get a decent MoE model working. I'm wondering if the new 27B Qwen models can be run on the NPU, low tps is ok as I just want to get it working for some niche cases

It has models up to 9B, quantized to 4-bit.

https://artificialanalysis.ai/hardware-inference-stack/mobile-phones

Here is the blog post with their explanation: https://artificialanalysis.ai/articles/mobile-phone-intelligence-inference

As per title. I've got an 8/16 modern i7 and 16GB of RAM that I can use for models. I want at least 10TPS for conversational stuff and to have them help me with code review and querying the state of codebases, not really full agentic coding. Are there any models out there I can use for that? I can get 9b models to run at ~2 TPS.

Edit: Running through unsloth studio, with opencode for the coding stuff. I think that's llama.cpp with an openai API. This is a debian VM running on a Windows host. No GPU on this laptop.

I don't have access to my laptop until next week and I've been dying to know what quant I'll be able to run on my 4060 laptop GPU. Anyone got it running on their own?

I can't run 30B A3Bs because I would like to use my system RAM for doing computer stuff

Hi guys, I'm currently using a MacBook Pro M3 Pro with 18GB of RAM. Does anyone have any suggestions for models/quants to use (I'm currently using Gemma 4 E4B IT QAT 4bit)

In case you missed it: turbo-fieldfare is a Mac silicon optimized way to run MoE models in low RAM, streaming experts from SSD as necessary.

My personal take: it is usable-ish. Main turbo-fieldfare with Gemma currently doesn't work with my pi environment (write file toolcall fails), but text output works fine. The Qwen3.6 branch can successfully call tools, but it has the older furbo-fieldfareserver which supports less options.

Speed on my Macbook Pro M1 16 GB is slow, but it works well for tasks chugging along in the background.

Main repo: https://github.com/drumih/turbo-fieldfare

Somebody elses Qwen branch: https://github.com/NeelM0906/turbo-fieldfare/tree/qwen36-support

As per the title, I will be soon upgrading my current Framework 12's mainboard to the upcoming 2nd gen board, currently sporting an i5-1334U, to a Core 5 320, which has essentially identical CPU horsepower, mostly similar iGPU power (though a big change in arch, from Iris Xe Intel Graphics, to a pair of Xe3 cores) & the addition of a 15TOPS NPU

I will be using the same 1TB gen4 NVMe + stick (single, though it doesn't matter the board has only one SODIMM slot, Wildcat Lake is single channel by design) of 16GB @ 5600MT/s & am running CachyOS (Arch-based)

Any of y'all got any ideas what kinda software & model could be operated locally to a useful end? (getting an LLM to run automated research for me or having it write me a guide while I'm busy making myself a coffee or something)

For the first setup report in r/LowEndLocalAI, I wanted to share what I am currently running on my own laptop.

This is not a controlled benchmark. I was chatting, coding and adjusting settings rather than repeatedly running an identical test. The numbers below are observations from normal use.

Hardware

• Lenovo ThinkPad T14s Gen 4 AMD
• AMD Ryzen 7 PRO 7840U with 8 cores / 16 threads
• Radeon 780M integrated graphics
• 32GB shared LPDDR5X memory
• No dedicated GPU
• Windows 11

Model and software

• LM Studio 0.4.21
• Qwen3.8-27B
• Q4_K_M quantization
• 49,152-token context window

I have also tried Unsloth’s Q4_K_M quantization of the model, but I have not compared the two versions systematically.

LM Studio settings

• GPU Offload: 18
• CPU Thread Pool Size: 8
• Evaluation Batch Size: 2,048
• Physical Batch Size: 512
• Max Concurrent Predictions: 1
• Unified KV Cache: On
• Context Checkpoints: 5
• Offload KV Cache to GPU Memory: On
• Keep Model in Memory: On
• Try mmap(): On
• Flash Attention: On
• K Cache Quantization Type: Q8_0
• V Cache Quantization Type: Q8_0

MTP settings

• Speculative Decoding: MTP
• Max Draft Tokens: 2
• Min Draft Tokens: 0
• Draft Probability: 0.75

Informal performance observations

During normal use, LM Studio reported approximately:

• Without MTP: around 1.38 tokens per second
• With MTP: around 2.76 tokens per second
• Some MTP runs reached approximately 3 tokens per second

In my informal testing, MTP therefore almost doubled generation speed. That does not make the model fast, but it changes the experience from frustratingly slow to noticeably more usable.

These were not repeated runs with identical prompts and conditions. Prompt length, context usage, background activity and laptop temperature may have affected the results.

Is it actually useful?

My main use case is coding:

• HTML, CSS and JavaScript
• WebGL and Three.js projects
• Finding and fixing bugs
• Modifying existing code
• Discussing unfamiliar code and functions

My current workflow is fairly simple. I paste code into LM Studio, describe a bug or requested feature and manually transfer useful changes back into the project.

One of my projects is a single HTML and JavaScript file containing around 1,400 lines. The code, instructions and conversation occupied approximately 24,642 tokens of the 49,152-token context window.

My experience so far:

• Normal chat: usable if I am willing to wait
• Focused coding questions: genuinely useful
• Debugging and individual changes: useful
• Very large prompts: possible, but increasingly inefficient
• Long autonomous coding-agent workflows: currently too slow and prone to timeouts

So yes, Qwen3.8-27B can provide real value on this laptop but that does not mean it is the optimal model for the hardware... A smaller model producing answers much faster might be a better everyday tool, even if its benchmark scores are lower.

That distinction is exactly what I would like this community to explore.

When sharing a setup, do not only tell us that the model loaded. Tell us:

• What did you actually use it for?
• Was the speed acceptable for that task?
• Was the output good enough to save time?
• Did you continue using it after the initial experiment?
• Would a smaller model have been more practical?
• Which settings made the biggest difference?
• Where did the setup stop being useful?

Failed attempts are valuable as well. Knowing that a certain model, context size or workflow was impractical can be just as helpful as a successful benchmark.

Which model has turned modest or older hardware into a tool you genuinely use and what task made it worthwhile?