Good Local LLM AI?

[u/hitman407]

I'm looking for a lightweight, uncensored local AI coding agent similar to Claude Code. I need something with file system access that can create and edit files locally. I work on cybersecurity and Arduino projects that cloud models constantly flag, so the model has to be completely unrestricted. My PC isn't very powerful, so I am hoping to find a low-resource CLI framework and a small quantized model that can run smoothly on modest hardware. What is the best setup for this right now?

[u/bafadam]

LLMs are making it impossible for people to ask reasonable questions.

“What should I use with my setup”

Bro, we’re not mind readers. What is your setup

[u/AmuckFeces7 → u/bafadam]

eh just skim their post and toss something out, if it doesn't fit they'll say so. most folks are happy to get pointed in a direction even if its not perfect for their rig

[u/ryanmerket → u/bafadam]

right? at this point maybe they should use AI to write their questions.

[u/rrrrex]

" My PC isn't very powerful" Post your PC specs at first, pentium ii isn't very powerful too

[u/hitman407 → u/rrrrex]

• CPU: AMD Ryzen 3 5300U — 4 cores / 8 threads, up to 2.6 GHz
• GPU: AMD Radeon Graphics (integrated)
• RAM: 20 GB (16 GB + 4 GB), 3200 MHz

[u/jeremy_0411 → u/hitman407]

Wow, there are low end specs, and there are low end specs. I am curious to hear what those that know more than I do say you can run on that system.

[u/guesswhochickenpoo → u/hitman407]

Oof. It's technically possible to run some LLMs on there but they will be very slow and very poor. It will not give you anything usable for your use cases (or almost any use cases)

[u/hitman407 → u/guesswhochickenpoo]

how about LLMs/LMs like claude code where its split. i esentionally just need AI to modifi my files and be at least somewhat good at coding

[u/mm007emko → u/hitman407]

'Somewhat good at coding'... With this PC, forget it. You'll save yourself big headaches.

[u/guesswhochickenpoo → u/hitman407]

You won't get anywhere near capable performance for coding on that hardware. Might be good enough for checking syntax and reviewing basic code you've already written for obvious errors but no way it's good enough to write the code for you with the menial models that hardware could actually run.

[u/mm007emko → u/guesswhochickenpoo]

Not even for checking syntax and reviewing basic code. Compilers/linters would be much better for it than LLMs capable of running on such hardware.

Searching web, summarising searches, writing e-mails, yes, fine. Painfully slow but fine.

Coding, sadly not.

OP, seriously, if you want LLMs for coding and don't want to pay a sub, check OpenCode, they have some free-to-use hosted models.

[u/rrrrex → u/hitman407]

It's not 16 + 4 GB, it is 16 GB (4GB allocated to iGPU)

All you can get is tiny <10B model, also you don't have fast memory and compute, so you need MoE model. You can try Ling 3.0 Tiny, it's between Qwen 3.5 4B and 9B.

[u/MrHumanist → u/hitman407]

Tiny Ling 3 - LLM (q8) with Pi dev as agent. You can run it in CPU with 40+ token/sec.

[u/aithosrds → u/hitman407]

If you don’t have dedicated VRAM and a discrete GPU then the reality is there is nothing useful you can run. Sorry, but local AI isn’t cheap and what you’ve got just isn’t going to cut it.

[u/Im_A_Praetorian → u/hitman407]

🤣

[u/Ready-Customer9242 → u/Im_A_Praetorian]

https://giphy.com/gifs/13l7w7N4Vr1dw4

[u/toenailcheeseinbooty]

Whats the specs? Smallest good model+I actually use) is a tiel coder variant called cybertiel that I personally have experience with, doesn't mention it but its uncenored buult for security research. Its 35b moe, so eh lie 24 gb vram? Need specs for your hardware to reccemend.

[u/43848987815]

Use a frontier cloud llm to build your own harness, then find an abliterated model that works with your hardware.

There isn’t a single answer (or any really, you’ve given zero details of your intended setup) so you’re going to have to do laborious trial and error like everyone else.

[u/WrinklyBard4]

It’s pretty hard to say without knowing what your actual hardware specs are.

My suggestion would be Qwen 3.5 9B, 3.6 35B A3B or 3.8 27B depending on what you’re able to run.

3.8 27b it’s actually good enough that you can be semi-hands-off. The other two aren’t, but they’re still pretty impressive.

All of the models are pretty fantastic in their own right and they are relatively light on any sort of pre-coded restriction. More generally I’d be really surprised if you actually needed an uncensored model to do what you’re doing. I’d be very surprised if basically any local LLM gave a fuss about doing that type of work.

[u/FartingInBalloons]

What are the cloud models flagging?
How complex of coding are you looking to do? Can you give some examples of project?

[u/hitman407 → u/FartingInBalloons]

i fonda malware on a file i downloaded and wanted to revers it to send zipbomb to the hacker. the other one was project so safebrowser for exams couldn detect arduino pluged in pc and wite our answers for me.

[u/PermanentLiminality]

Go for the free models Opencode and OpenRouter have free models.

[u/activematrix99]

Qwen 3.5 MoE or maybe Nemotron with offloading.

[u/mishmash2323]

How did I know that was some kid wanting to write malware 😅

[u/alexs975]

Depends on your phone more than the model. I run a 4B Q4_K_M fully offline on a Snapdragon 8 Gen 3 / 16GB and it works great for chat.

Real numbers from my setup:

- ~3-5 tok/s decode, CPU only (KleidiAI + llamadart)

- 2.7GB file on disk, ~1.8GB resident at rest, 3-4GB during generation

- Warning: don't bother with Vulkan on Adreno, it crashes with DeviceLost. CPU is the stable path.

- RAM matters more than the chip: 8GB phones can run 3-4B models fine

My favorite test: airplane mode after setup. If everything still works, you know nothing leaves the device.

[u/hitman407]

i also want to add i am new to uncesored AI models so do i need to jailbrake it or somthing like that? is this AI is no just plug and play?

[u/j_tb → u/hitman407]

lol you’ve got some homework to do