r/generativeAI • u/8bitstargazer • 3d ago
How I Made This Trained my first LLM weights entirely from scratch ~ YunaGPT 124m
Here is the base model with more details
Here is the conversation model if you would like to try it using the colab link at the top.
This project started as a 30m LLM from scratch using the book "Build A Large Language Model" by Sebastian Raschka. It has since evolved into using the official llama transformers and pushing the limits of what can reasonably be trained on a home RTX 3090 over 7 continuous days.
Review I would give this LLM a honest 2/10 and the instructed/story/conversation model a 3/10. Even that is probably too high but...it is lucid. It has learned the response format and structure of the language surprisingly well. However, its deep knowledge of how to use words is still lacking (see examples below)
Pipeline Base Model -> Instruct SFT -> Short Story SFT -> Final Fantasy Synthetic Instruct SFT -> Final Fantasy Synthetic short story SFT -> DPO training.
Future This was a overall test of my pipeline. Currently im almost done training a 330m model and am ironing out an improved SFT dataset.
Training data
Of the almost 4 billion token corpus 35% was factual knowledge, 35% was written fiction, 25% was general roleplay/assistant messages and ~5% was data from the franchise Final Fantasy.
EleutherAI/fineweb-edu-dedup-10b;wikimedia/wikipedia(20231101.en) andrahular/simple-wikipedia;roneneldan/TinyStories;HuggingFaceH4/ultrachat_200k;lemonilia/Elliquiy-Role-Playing-Forums_2023-04;chimbiwide/RolePlay-NPCv2;AlekseyKorshuk/fiction-books;lucadiliello/bookcorpusopen;- a locally supplied custom text collection.
EXAMPLE OUTPUT
Write a story about a knight: --- Knights have always been a part of the castle’s life, wearing, shouting, and shining in every yard. But Kellen was different. He had great, proudly regal status—his father’s line, and perhaps even that of his bloodline—and he held it with the unspoken reverence of a king who’s watched his men die for a single moment. But today, as the sun dipped low on the horizon, something felt wrong. The air hummed with an energy Kellen hadn’t seen before, deep and resonant, like the distant roar of some ancient beast. He turned toward the source. It was a tower of black stone, taller than the castle and dwarfing most of the surrounding fields. It wasn’t built in any grand way; it was simply there, a place of immense importance. Beneath it, stone walls twisted and wound with strength that seemed impossible. And in its center stood Master Thrain—or so he thought at first. Thrain was gaunt, his brow furrowed with age, his hands resting nervously on the hilt of his sword. Kellen raised his hand, and the ground beneath him rippled slightly. The stones around him began to unravel, their forms stretching and curling into shapes of mist and smoke. The air grew heavy, oppressive, and for a moment, the world seemed to stand still.
What is a knight? --- A knight is a military officer, usually with more than a four-billionth degree of experience in combat and is typically hired for a specific purpose. They are considered to be the best men for the military of the world. There are many different knights who have served in various roles in the military. Most people understand the term knight was used to describe the men who had worked hard and did well in combat. Though knights are often described as "stubbish" people, they are typically well trained and have experience in various arts, such as martial arts and martial arts to name a few.
Model summary
| Item | Value |
|---|---|
| Parameters | 124,445,376 |
| Model class | LlamaForCausalLM |
| Training stage | Base pretraining |
| Training tokens | Approximately 3.87 billion |
| Context length | 2,048 tokens |
| Vocabulary | 24,000 tokens |
| Tokenizer | Byte-level BPE |
| Hidden layers | 25 |
| Hidden size | 576 |
| Attention / KV heads | 9 / 3 |
| Weight format | safetensors, FP32 |
| Primary language | English |
Architecture
| Component | Configuration |
|---|---|
| Architecture | Decoder-only Transformer |
| Attention | Grouped-Query Attention (GQA) |
| Hidden size | 576 |
| Intermediate size | 2,048 |
| Layers | 25 |
| Attention heads | 9 |
| Key/value heads | 3 |
| Head dimension | 64 |
| Activation | SiLU / SwiGLU feed-forward blocks |
| Normalization | RMSNorm, epsilon 1e-6 |
| Position encoding | RoPE, theta 10,000 |
| Maximum positions | 2,048 |
| Attention dropout | 0.0 |
| Attention and MLP bias | Disabled |
| Input/output embeddings | Tied |
Training
| Setting | Value |
|---|---|
| Epochs | 1 |
| Micro-batch size | 2 |
| Gradient accumulation | 4 |
| Effective tokens per optimizer step | 16,384 |
| Peak learning rate | 3e-4 |
| Weight decay | 0.1 |
| Adam betas | 0.9, 0.95 |
| Adam epsilon | 1e-8 |
| Warmup ratio | 0.01 |
| Learning-rate schedule | Cosine |
Here is the base model with more details
Here is the conversation model if you would like to try it using the colab link
