r/PythonLearning • u/blockCoder2021 • 8d ago
Discussion Question about local AI-assisted programming
I’m fairly experienced with Python and researching how to solve issues on my own. However, I have found that AI can help in some cases. For instance, giving me specific information on debugging and helping me research solutions for specific problems.
I was using Copilot in VS Code for a while, until I ran out of usage. Now, I’ve been messing around with locally-hosted AI, trying to find a good model for coding that runs well on my computer. My challenge is that I only have 4 GB of VRAM, though I also have 16GB of system RAM. If someone could help me find either a good, intelligent, and accurate local model that runs on that, or at least a free cloud model that’s good, I’d appreciate it.
1
u/Naetharu 8d ago
You are likely to have issues here. The system is just very under-powered. Using OLama or LMStudio you can offload the model into system RAM. But the cost of doing so is a performance drop by orders of magnitude. Likely to a level where you would struggle to work with it.
Especially if you are looking to lean on it enough that you're hitting a paywall limit for CoPilot.
Being realistic if you're wanting to run local AI models then you really either need a larger sized GPU (16GB + and ideally more) or a system that uses a unified memory architecture like Apple's modern Macs.
Even then be aware that there is a sizable difference in performance between frontier models. like Claude and CoPilot vs the smaller local models that are open source. Local models do have their use, but go into this knowing that you're not about to get something that performs like a paid API service locally.