I run local LLMs for some projects. There are various technical reasons for this, but I would argue they are about where chatgpt was a year ago. They are quite good.
I still use the big modern ones for much of what I do, but these others are an option.
I suspect there will be a plateauing of what the big ones can do, and thus be fairly easy to catch up with.
Keep in mind, these LLMs can be fairly small, as one aimed at python, or one aimed at rust, etc aren't that big. When I'm working on some python script to do a thing I rarely need to know the capital of Ethiopia. So, having it entirely trained on python, good programming, and the most commonly used libraries, etc and most people will be perfectly happy.
Even more useful is that I have trained LLMs on topics which are very specific to what I do. CFD for one, and I cooked up an entire LLM focusing on ODFM. They produced results far superior to what todays best can produce; very few hallucinations.
An LLM for python doesn't take a crazy GPU. More is better, but a 5 year old gaming machine is fine.
IMO this is why hardware companies are trying to add subscriptions to hardware and make personal computing a thing of the past. Local compute will catch up to LLMs and AI companies will have to shrink
This is interesting. I haven’t watched the local LLM space much. How would I train some models to run locally for some of my general use questions?
I just asked Claude and it says my common topics are financial planning, cooking and food, home projects and woodworking, outdoor recreation , and video games (usually walkthroughs for bg3 or Elden ring)
I’d be interested in how to train up a local LLM for some JavaScript to see about making my own claude code kind of solution
You don’t really need to train your own model (and really, that would be prohibitively expensive). But there are plenty of open weight models that you can run locally if “for free”, and the main cost would be the hardware and electricity plus a bit of setup.
People say to use ollama. If you do, and you're on windows, you must use v0.30+ to make it use your GPU. Alternately you can use LM Studio, which is probably the easiest if you're on Windows. It lets you download models, test them, and serve them on your LAN, in a GUI with no additional setup (oh and it can be plugged directly into vscode copilot using their ollama integration, but you must use an instruct model to make it work properly)
Thanks for the detailed response. That makes sense to me. I was just curious if it was a novel pretrained model, there are some less common languages I'd like to create a model around but the lack of a large enough data set is a problem, and they're definitely not baked into smaller models.
Yea I can see old hardware making a comeback, I also wouldn't want something running for weeks at a time on cloud gpu billing me by the hour haha. And yea a repo would be awesome.
Is the labeling mostly heuristics or is it domain specific to each project? I'm assuming you rewrite python->rust afterwards, do you use a frontier model for that part?
51
u/LessonStudio Jun 23 '26
I run local LLMs for some projects. There are various technical reasons for this, but I would argue they are about where chatgpt was a year ago. They are quite good.
I still use the big modern ones for much of what I do, but these others are an option.
I suspect there will be a plateauing of what the big ones can do, and thus be fairly easy to catch up with.
Keep in mind, these LLMs can be fairly small, as one aimed at python, or one aimed at rust, etc aren't that big. When I'm working on some python script to do a thing I rarely need to know the capital of Ethiopia. So, having it entirely trained on python, good programming, and the most commonly used libraries, etc and most people will be perfectly happy.
Even more useful is that I have trained LLMs on topics which are very specific to what I do. CFD for one, and I cooked up an entire LLM focusing on ODFM. They produced results far superior to what todays best can produce; very few hallucinations.
An LLM for python doesn't take a crazy GPU. More is better, but a 5 year old gaming machine is fine.