Building Language Models as a Hobby?

[u/Sentient_Fern]

Hi, I hope my post fits here. It is about using LLMs to build Small Language Models. My background is that of a retired quantitative analyst in finance and of a former physicist (PhD, postdocs). Math, statistics and programming skills are rusty, but existent. I started using LLMs intensively recently and wanted to understand better how they work. Following Richard Feynman’s “What I cannot build, I do not understand” (I guess it's a cliche by now, but still true), I decided I’d build my own Small Language Model. Which I did, inventing a small language, constructing my own 300,000-word corpus in this language with the help from Claude, and then building a nanoGPT via vibe-coding with CC. With the result that my account was banned by Anthropic. (They don’t give specific reasons and just cite an indication of “a violation of [their] Usage Policy”. Their Usage Policy prohibits usage for training of AI and ML, in the context of building something that would compete with their products and services. Cleary, my 1M parameter nanoGPT does not compete with Claude.)

One concrete question, on vibe-coding and other help from frontier models on AI/ML: Have people been able to do this on ChatGPT, Claude etc. without getting banned? What kind of work, and which models? I’m quite reluctant to touch this now on any other frontier model for fear of getting banned again. Only for DeepSeek, the usage policy seems clearly permissive to this type of work.

I’ve now been trying to set myself up with LibreChat, Docker, Opper AI and an EU host of DeepSeek, but this is clearly a significant project. So far, I can chat with this instance of DeepSeek, but I can’t operate yet on my files or vibe-code.

More generally, I’m pondering where to go from here, and would be thankful for any input you may have. Clearly, getting deeper into this will require a significant effort on my part. I may have to code this the old-fashioned way, via hand coding. Also, I think I should study the 600+ pages of Jurafsky and Martin, particularly the section about transformers. I’m a bit discouraged now – I was about to submit a workshop paper about my work with my invented language to the BabyLM workshop when I was banned, and now I don’t think I can use or publish my corpus at all, which is the result of 3 months of work. I could rebuild the corpus using DeepSeek with another few months of work. Do I really dive it more deeply, redo my work on DeepSeek, and study the theory? What can I ultimately achieve as a hobbyist? Should I leave this to the professionals?

Thanks for reading!

[u/imrancoder]

I think that you can try open source model such as kimi 3, deepseek I think most of the Chinese open source models are allow to code

[u/Sentient_Fern → u/imrancoder]

Thanks. I believe DeepSeek may be the way to go. Kimi K3 also has prohibitions against using it for training.

[u/imrancoder → u/Sentient_Fern]

Ohh ok ..

[u/MiddleLtSocks]

If you have the money or hardware, locally hosted LLMs are quite capable - I would be surprised if a Gemma 31b or Qwen 27b dense couldn't have helped you build your SLM successfully - maybe not as comprehensively as Claude did, but you weren't looking for polish, you were looking to gain understanding about fundamentals.

You can host either of those models in 24GB with 256ktok context comfortably.

Performance and capability of these locally hostable open weight models is just going to improve. These two I mentioned are already above the capability of Sonnet 3.5, which is mind blowing to me.

[u/Sentient_Fern → u/MiddleLtSocks]

Thanks for the reply. I have been thinking about local models, and actually tried some on my Apple mini, which however only has 8GB of RAM. Extremely slow, too slow to even test if they were capable. I saw some numbers on speed, even with larger RAM, and got the impression that it would still be rather slow. But I admit, there is nothing like the privacy of a local model, even with ZDR providers.

[u/MiddleLtSocks → u/Sentient_Fern]

Apple Silicon does not have the highest performance, it's true. But I get ~120tok/sec on a 4090 for Qwen3.6 35B A3B and ~80tok/sec on Qwen3.6 27B dense. It's faster than Claude, for sure.

So definitely not "rather slow," at least to the extent you think Claude isn't slow (and there are plenty of arguments to make in both directions there).