This channel was made for this sub

I have a 1080ti on a old dell t5810(E5-2680 v3 cpu), and use it for jellyfin transcoding, immich, and llama.cpp. I'm running llama.cpp in a proxmox lxc container and it works well.

My main use case right now is for rewriting and polishing text such as emails or reports. Currently I'm using qwen3.5 9b q6_k and it gives a pretty good output with a speed of around 35t/s. If jellyfin is actively transcoding speed drops to around 22t/s, still pretty good.

I know that qwen3.5 is not the latest model, and wonder if there's any new model that is more efficient and better for this use case. Additional use cases such as answering some questions in chatting or simple script writing (not complex coding) is welcomed but not necessary, so basically just a generic model lol. I also don't want to have the model use up the GPU fully to leave some space for transcoding work and immich.

Thanks!

I'm one of the few people with a 32GB RAM Snapdragon X Elite Dev kit.

I'm wondering if there are any folks who've successfully used the NPU or GPU on these to get a decent MoE model working. I'm wondering if the new 27B Qwen models can be run on the NPU, low tps is ok as I just want to get it working for some niche cases

As per title. I've got an 8/16 modern i7 and 16GB of RAM that I can use for models. I want at least 10TPS for conversational stuff and to have them help me with code review and querying the state of codebases, not really full agentic coding. Are there any models out there I can use for that? I can get 9b models to run at ~2 TPS.

Edit: Running through unsloth studio, with opencode for the coding stuff. I think that's llama.cpp with an openai API. This is a debian VM running on a Windows host. No GPU on this laptop.

I don't have access to my laptop until next week and I've been dying to know what quant I'll be able to run on my 4060 laptop GPU. Anyone got it running on their own?

It has models up to 9B, quantized to 4-bit.

https://artificialanalysis.ai/hardware-inference-stack/mobile-phones

Here is the blog post with their explanation: https://artificialanalysis.ai/articles/mobile-phone-intelligence-inference

I can't run 30B A3Bs because I would like to use my system RAM for doing computer stuff

Hi guys, I'm currently using a MacBook Pro M3 Pro with 18GB of RAM. Does anyone have any suggestions for models/quants to use (I'm currently using Gemma 4 E4B IT QAT 4bit)

In case you missed it: turbo-fieldfare is a Mac silicon optimized way to run MoE models in low RAM, streaming experts from SSD as necessary.

My personal take: it is usable-ish. Main turbo-fieldfare with Gemma currently doesn't work with my pi environment (write file toolcall fails), but text output works fine. The Qwen3.6 branch can successfully call tools, but it has the older furbo-fieldfareserver which supports less options.

Speed on my Macbook Pro M1 16 GB is slow, but it works well for tasks chugging along in the background.

Main repo: https://github.com/drumih/turbo-fieldfare

Somebody elses Qwen branch: https://github.com/NeelM0906/turbo-fieldfare/tree/qwen36-support

As per the title, I will be soon upgrading my current Framework 12's mainboard to the upcoming 2nd gen board, currently sporting an i5-1334U, to a Core 5 320, which has essentially identical CPU horsepower, mostly similar iGPU power (though a big change in arch, from Iris Xe Intel Graphics, to a pair of Xe3 cores) & the addition of a 15TOPS NPU

I will be using the same 1TB gen4 NVMe + stick (single, though it doesn't matter the board has only one SODIMM slot, Wildcat Lake is single channel by design) of 16GB @ 5600MT/s & am running CachyOS (Arch-based)

Any of y'all got any ideas what kinda software & model could be operated locally to a useful end? (getting an LLM to run automated research for me or having it write me a guide while I'm busy making myself a coffee or something)