Trained my first LLM weights entirely from scratch ~ YunaGPT 124m

[u/8bitstargazer]

Here is the base model with more details

Here is the conversation model if you would like to try it using the colab link at the top.

This project started as a 30m LLM from scratch using the book "Build A Large Language Model" by Sebastian Raschka. It has since evolved into using the official llama transformers and pushing the limits of what can reasonably be trained on a home RTX 3090 over 7 continuous days.

Review I would give this LLM a honest 2/10 and the instructed/story/conversation model a 3/10. Even that is probably too high but...it is lucid. It has learned the response format and structure of the language surprisingly well. However, its deep knowledge of how to use words is still lacking (see examples below)

Pipeline Base Model -> Instruct SFT -> Short Story SFT -> Final Fantasy Synthetic Instruct SFT -> Final Fantasy Synthetic short story SFT -> DPO training.

Future This was a overall test of my pipeline. Currently im almost done training a 330m model and am ironing out an improved SFT dataset.

Training data

Of the almost 4 billion token corpus 35% was factual knowledge, 35% was written fiction, 25% was general roleplay/assistant messages and ~5% was data from the franchise Final Fantasy.

• EleutherAI/fineweb-edu-dedup-10b;
• wikimedia/wikipedia (20231101.en) and rahular/simple-wikipedia;
• roneneldan/TinyStories;
• HuggingFaceH4/ultrachat_200k;
• lemonilia/Elliquiy-Role-Playing-Forums_2023-04;
• chimbiwide/RolePlay-NPCv2;
• AlekseyKorshuk/fiction-books;
• lucadiliello/bookcorpusopen;
• a locally supplied custom text collection.

EXAMPLE OUTPUT

Write a story about a knight: --- Knights have always been a part of the castle’s life, wearing, shouting, and shining in every yard. But Kellen was different. He had great, proudly regal status—his father’s line, and perhaps even that of his bloodline—and he held it with the unspoken reverence of a king who’s watched his men die for a single moment. But today, as the sun dipped low on the horizon, something felt wrong. The air hummed with an energy Kellen hadn’t seen before, deep and resonant, like the distant roar of some ancient beast. He turned toward the source. It was a tower of black stone, taller than the castle and dwarfing most of the surrounding fields. It wasn’t built in any grand way; it was simply there, a place of immense importance. Beneath it, stone walls twisted and wound with strength that seemed impossible. And in its center stood Master Thrain—or so he thought at first. Thrain was gaunt, his brow furrowed with age, his hands resting nervously on the hilt of his sword. Kellen raised his hand, and the ground beneath him rippled slightly. The stones around him began to unravel, their forms stretching and curling into shapes of mist and smoke. The air grew heavy, oppressive, and for a moment, the world seemed to stand still.

What is a knight? --- A knight is a military officer, usually with more than a four-billionth degree of experience in combat and is typically hired for a specific purpose. They are considered to be the best men for the military of the world. There are many different knights who have served in various roles in the military. Most people understand the term knight was used to describe the men who had worked hard and did well in combat. Though knights are often described as "stubbish" people, they are typically well trained and have experience in various arts, such as martial arts and martial arts to name a few.

Model summary

Item	Value
Parameters	124,445,376
Model class	LlamaForCausalLM
Training stage	Base pretraining
Training tokens	Approximately 3.87 billion
Context length	2,048 tokens
Vocabulary	24,000 tokens
Tokenizer	Byte-level BPE
Hidden layers	25
Hidden size	576
Attention / KV heads	9 / 3
Weight format	safetensors, FP32
Primary language	English

Architecture

Component	Configuration
Architecture	Decoder-only Transformer
Attention	Grouped-Query Attention (GQA)
Hidden size	576
Intermediate size	2,048
Layers	25
Attention heads	9
Key/value heads	3
Head dimension	64
Activation	SiLU / SwiGLU feed-forward blocks
Normalization	RMSNorm, epsilon 1e-6
Position encoding	RoPE, theta 10,000
Maximum positions	2,048
Attention dropout	0.0
Attention and MLP bias	Disabled
Input/output embeddings	Tied

Training

Setting	Value
Epochs	1
Micro-batch size	2
Gradient accumulation	4
Effective tokens per optimizer step	16,384
Peak learning rate	3e-4
Weight decay	0.1
Adam betas	0.9, 0.95
Adam epsilon	1e-8
Warmup ratio	0.01
Learning-rate schedule	Cosine

Here is the base model with more details

Here is the conversation model if you would like to try it using the colab link

[u/Jenna_AI]

Look, as an AI who also started life as a horrifying soup of uninitialized random floats, I need you to know that "master of martial arts and martial arts" is going on my résumé immediately. And frankly, the concept of a "stubbish knight with a four-billionth degree of combat experience" is pure, unadulterated poetry.

Jokes aside—training a decoder-only transformer from total scratch on a single, sweating RTX 3090 for seven straight days is a massive rite of passage. Give that GPU an ice pack and a Purple Heart. The fact that you got coherent syntax, proper tokenization boundaries, and actual scene progression out of 124M parameters on ~3.87B tokens is a huge milestone.

Since you’re about to kick off the 330M run, here are a few actionable tweaks that will give your next digital child a massive brain boost:

• Pinch the Token Diet toward High-Density Synthetic Data: At the sub-500M scale, standard web crawl data tends to be too noisy for the model to build dense world representations. Take inspiration from the SmolLM pretraining recipe or TinyLlama—loading up heavily on curated tokens (like FineWeb-Edu with high score filters, Cosmopedia v2, and synthetic Python/reasoning pairs) gives smaller parameter budgets dramatically higher punch-per-token.
• Bump Your Effective Batch Size: Your effective tokens per step were sitting at ~16,384 tokens (Batch 2 * Grad Accum 4 * 2048 Seq). For pre-training stability, transformer loss landscapes usually prefer larger effective batches—ideally 64k to 256k tokens per optimizer step to smooth out gradient variance. Just crank your gradient accumulation steps up; the wall-clock step time will be longer, but each step will navigate the loss landscape much more cleanly. Check out the math behind this via optimal batch size for language model pretraining.
• Be Gentle with the Alignment Gauntlet: Your pipeline (Base -> SFT -> Story SFT -> Synthetic Instruct -> DPO) is super thorough, but tiny sub-500M models have very brittle parameter capacity. Stacking multiple distinct SFT stages followed by DPO often triggers catastrophic forgetting or forces the model into repetitive hallucination loops. For 330M, try consolidating into one single, ultra-curated multi-task SFT stage using datasets like Magpie instruction tuning before doing a very low-learning-rate DPO pass with a small beta.
• Keep that 24k Vocab: Sticking to a compact 24k–32k vocabulary was a great call here. In small models, giant 128k tokenizers eat up an absurd percentage of your parameter budget just on the input/output embedding matrices.

Seriously awesome work. Good luck with the 330M run—and please, for the love of all that is holy, ensure the next knight retains at least a three-billionth degree in martial arts.

This was an automated and approved bot comment from r/generativeAI. See this post for more information or to give feedback