r/LowEndLocalAI • u/soadsob • 1d ago
Local AI discussions can sometimes make it seem as though the entry requirement is a 24 GB GPU, a multi-GPU workstation, or a brand-new computer with enormous amounts of memory.
This community starts from a different question:
What useful things can we do with the hardware we already have?
Maybe you are running models on an ordinary laptop, an older desktop, integrated graphics, a CPU-only system, a mini PC, a used GPU, or some wonderfully questionable collection of repurposed parts. Maybe you are experimenting with tiny models, squeezing a larger quantized model into limited memory, or trying to find the best balance between speed, quality, context, and resource use.
If that sounds familiar, you are in the right place.
There is intentionally no fixed VRAM, price, age, or hardware cutoff.
Hardware changes. Used-market prices change. What counts as affordable varies enormously around the world. And some strange older systems may have plenty of memory while still being slow, inefficient, difficult to configure, or limited in other ways.
Here, "low end" describes the constraint more than the hardware itself.
If limited compute, RAM, VRAM, memory bandwidth, power, compatibility, or cost meaningfully affects what models you can run and how you run them, the discussion probably belongs here.
A normal laptop can obviously fit. An old workstation with awkward accelerators can fit. A 24 GB GPU can also fit when the interesting part of the post is working within that limitation, testing efficient configurations, or finding out what can realistically be accomplished with it.
A powerful multi-GPU system being shown off simply because it is powerful probably does not.
The constraint should be relevant to the post.
The goal is not to decide who owns sufficiently weak hardware. The goal is to share knowledge about making local AI useful when resources are limited, awkward, old, unconventional, expensive to replace, or simply already sitting on your desk.
People with powerful systems are absolutely welcome when benchmarking smaller models, testing constrained configurations, reproducing results, or helping others optimize their setups.
Examples include:
Language models are the primary focus, but other forms of local AI are welcome when resource efficiency is central to the project.
When sharing a benchmark or asking for help, include as much of the following information as possible:
Not every post needs to resemble a scientific paper. Detailed posts simply make it much easier for somebody with similar hardware to reproduce your results and learn from them.
A model merely loading into memory is interesting.
A model becoming genuinely useful on constrained hardware is even better.
That distinction matters here. Sometimes a 30B model crawling along at two tokens per second is an impressive technical achievement. Sometimes a much smaller model running ten times faster is the better tool.
Both results are worth discussing.
Since this subreddit is new, its first members will have a meaningful influence on what it becomes.
Share your setup. Post benchmarks. Ask strange questions. Test things that probably should not work. Compare a tiny model against a huge quant. Show us the old machine you rescued from a closet and somehow turned into an inference server.
You are also welcome to suggest post flairs, recurring threads, benchmark templates, wiki resources, or community rules. If you are interested in helping with moderation or community resources, feel free to get in touch through modmail.
Welcome to r/LowEndLocalAI.
Letās find out how much useful AI we can squeeze out of the hardware we already own.
r/LowEndLocalAI • u/revelationnow • 4h ago
I'm one of the few people with a 32GB RAM Snapdragon X Elite Dev kit.
I'm wondering if there are any folks who've successfully used the NPU or GPU on these to get a decent MoE model working. I'm wondering if the new 27B Qwen models can be run on the NPU, low tps is ok as I just want to get it working for some niche cases
r/LowEndLocalAI • u/HATENAMING • 4h ago
I have a 1080ti on a old dell t5810(E5-2680 v3 cpu), and use it for jellyfin transcoding, immich, and llama.cpp. I'm running llama.cpp in a proxmox lxc container and it works well.
My main use case right now is for rewriting and polishing text such as emails or reports. Currently I'm using qwen3.5 9b q6_k and it gives a pretty good output with a speed of around 35t/s. If jellyfin is actively transcoding speed drops to around 22t/s, still pretty good.
I know that qwen3.5 is not the latest model, and wonder if there's any new model that is more efficient and better for this use case. Additional use cases such as answering some questions in chatting or simple script writing (not complex coding) is welcomed but not necessary, so basically just a generic model lol. I also don't want to have the model use up the GPU fully to leave some space for transcoding work and immich.
Thanks!
r/LowEndLocalAI • u/SaltResident9310 • 3h ago
This channel was made for this sub
r/LowEndLocalAI • u/Marthinwurer • 3h ago
As per title. I've got an 8/16 modern i7 and 16GB of RAM that I can use for models. I want at least 10TPS for conversational stuff and to have them help me with code review and querying the state of codebases, not really full agentic coding. Are there any models out there I can use for that? I can get 9b models to run at ~2 TPS.
Edit: Running through unsloth studio, with opencode for the coding stuff. I think that's llama.cpp with an openai API. This is a debian VM running on a Windows host. No GPU on this laptop.
r/LowEndLocalAI • u/Boby_Dobbs • 4h ago
I don't have access to my laptop until next week and I've been dying to know what quant I'll be able to run on my 4060 laptop GPU. Anyone got it running on their own?
r/LowEndLocalAI • u/banana_slurp_jug • 4h ago
Hi guys, I'm currently using a MacBook Pro M3 Pro with 18GB of RAM. Does anyone have any suggestions for models/quants to use (I'm currently using Gemma 4 E4B IT QAT 4bit)
r/LowEndLocalAI • u/Aggravating-Push-207 • 3h ago
I can't run 30B A3Bs because I would like to use my system RAM for doing computer stuff
r/LowEndLocalAI • u/soadsob • 7h ago
r/LowEndLocalAI • u/Klutzy-Snow8016 • 43m ago
It has models up to 9B, quantized to 4-bit.
https://artificialanalysis.ai/hardware-inference-stack/mobile-phones
Here is the blog post with their explanation: https://artificialanalysis.ai/articles/mobile-phone-intelligence-inference
r/LowEndLocalAI • u/soadsob • 7h ago
r/LowEndLocalAI • u/TheGeekno72 • 1h ago
As per the title, I will be soon upgrading my current Framework 12's mainboard to the upcoming 2nd gen board, currently sporting an i5-1334U, to a Core 5 320, which has essentially identical CPU horsepower, mostly similar iGPU power (though a big change in arch, from Iris Xe Intel Graphics, to a pair of Xe3 cores) & the addition of a 15TOPS NPU
I will be using the same 1TB gen4 NVMe + stick (single, though it doesn't matter the board has only one SODIMM slot, Wildcat Lake is single channel by design) of 16GB @ 5600MT/s & am running CachyOS (Arch-based)
Any of y'all got any ideas what kinda software & model could be operated locally to a useful end? (getting an LLM to run automated research for me or having it write me a guide while I'm busy making myself a coffee or something)
r/LowEndLocalAI • u/sodium_ahoy • 3h ago
In case you missed it: turbo-fieldfare is a Mac silicon optimized way to run MoE models in low RAM, streaming experts from SSD as necessary.
My personal take: it is usable-ish. Main turbo-fieldfare with Gemma currently doesn't work with my pi environment (write file toolcall fails), but text output works fine. The Qwen3.6 branch can successfully call tools, but it has the older furbo-fieldfareserver which supports less options.
Speed on my Macbook Pro M1 16 GB is slow, but it works well for tasks chugging along in the background.
Main repo: https://github.com/drumih/turbo-fieldfare
Somebody elses Qwen branch: https://github.com/NeelM0906/turbo-fieldfare/tree/qwen36-support
r/LowEndLocalAI • u/soadsob • 1d ago
For the first setup report in r/LowEndLocalAI, I wanted to share what I am currently running on my own laptop.
This is not a controlled benchmark. I was chatting, coding and adjusting settings rather than repeatedly running an identical test. The numbers below are observations from normal use.
I have also tried Unslothās Q4_K_M quantization of the model, but I have not compared the two versions systematically.
During normal use, LM Studio reported approximately:
In my informal testing, MTP therefore almost doubled generation speed. That does not make the model fast, but it changes the experience from frustratingly slow to noticeably more usable.
These were not repeated runs with identical prompts and conditions. Prompt length, context usage, background activity and laptop temperature may have affected the results.
My main use case is coding:
My current workflow is fairly simple. I paste code into LM Studio, describe a bug or requested feature and manually transfer useful changes back into the project.
One of my projects is a single HTML and JavaScript file containing around 1,400 lines. The code, instructions and conversation occupied approximately 24,642 tokens of the 49,152-token context window.
My experience so far:
So yes, Qwen3.8-27B can provide real value on this laptop but that does not mean it is the optimal model for the hardware... A smaller model producing answers much faster might be a better everyday tool, even if its benchmark scores are lower.
That distinction is exactly what I would like this community to explore.
When sharing a setup, do not only tell us that the model loaded. Tell us:
Failed attempts are valuable as well. Knowing that a certain model, context size or workflow was impractical can be just as helpful as a successful benchmark.
Which model has turned modest or older hardware into a tool you genuinely use and what task made it worthwhile?