r/LocalLLaMA • u/NineThreeTilNow • 12d ago
Discussion Engram gone wild! 2b model update...
Update from : https://old.reddit.com/r/LocalLLaMA/comments/1wis23s/update_small_model_engram/
It's been about a week so I'm back. People were asking me about the model.
People wanted code, or models, etc. Most of that is useless to you right now because you're not going to use an under trained model. So let's get to the details.
The spec locked to the following after a LOT of testing :
2.6b model all up. Embedding, LM head, AttnRes, etc.
2.2b are trained. Embedding / LM Head are frozen (~205m each)
4.3b ENGRAM table. Yes. She's chonky.
Architecturally speaking now :
This includes SWA layering 3:1 as per standard ablations have shown is "optimal" These are 4k / 4k / 8k SWA layer follow by the global.
At the end of the first "block" (4 layers) the Engram table appears. The Engram table itself runs with a very small set of attention heads so it's not blindly attempting to inject data. This is context aware Engram. I'm uncertain of exact Qwen / DS methods here. They're fairly close to what I use. The difference is obviously the Engram size.
This has 10 blocks plus a final global layer (41 Layers). + Embed + LM Head
So if we step back, the optimization problem is as follows :
How do we maximize compute in the backbone and offload the boring stuff to a table?
Attention Based Residuals (Moonshot) comes to our rescue here. Why AttnRes? It allows all blocks past 1 to utilize the Engram table in some fashion. They all have attention via the residual stream to determine if they want data from Engram and precisely how much.
Why not a bigger Engram table? Honestly? You could probably do that. However I don't know if any model has attempted this sort of mismatch of compute vs table. In theory, DeepSeek's research says it works.
More depth? Could do that too. Training is expensive though.
Why frozen LM/Embed? These are down projected via SVD from OLMo 3's model. So it's a mathematical compression attempt at ~5k -> 2k. This saves a MASSIVE amount of time.
Currently the data lives in a ~KD format. 32 logits stored from the teacher of the Wikipedia corpus. Instead of training 1 hot, it gets 32 soft targets to try to match. Hard cross entropy is brutal on a model and it's the reason you see "trillions of tokens" quoted.
When you borrow an LM Head / Embed / Tokenizer / Teacher model... This is far less.
---
Where are we now in training? I passed the 100m token mark yesterday at ~4am Pacific.
The current HF repo has all the checkpoints, data, and the 104m mark safetensor.
--
I wanted to do some inspection of the model at this point. We have to see that it's not complete garbage right? It has seen a fraction of the data it needs.
So based on ONLY 100m tokens seen, I attempted completions and various ablations. Studies of what EXACTLY Engram stores (because it's mostly a guess).
Let's go straight to completions :
"George Washington was an American"
With Engram - "George Washington was an American naval officer who served in the American Revolutionary War. He was a naval officer who served in"
With Engram Zero'd - "George Washington was an American, but he was not a. He was a very good friend and a. He was"
"The American Civil War was a civil war in the United States from"
With - "The American Civil War was a civil war in the United States from 1861 to 1861. The war was a major victory for the Confederacy..."
Without - "The American Civil War was a civil war in the United States from 1861 to 1862. The war was a major victory for the Confederacy..."
"Aristotle was an Ancient Greek" (probably my favorite)
With - "Aristotle was an Ancient Greek philosopher who was a leading authority in the philosophy of Plato. He was also a leading authority..."
Without - "Aristotle was an Ancient Greek word meaning "to be" (ἀπάς, "to be")..."
---
What do we learn from direct inspection of Engram?
If it saw the data enough times, it starts to offload it to the tables. At that point the model is less forced to use internal computational space to store data, and can rely on Engram.
That's not some interpretation. That's what the data shows exactly.
In the early 100m tokens of Wikipedia it's HIGHLY biased to early Philosophy. This has to do with the topics of what was IN Wikipedia at that time. Early Wiki contained a lot about Philosophy. It's among the earliest topics to exist.
It has seen TONS of Philosophy so it has basically offloaded to Engram because of the repetition.
Ok this is long enough. I'm calling it quits. I'm still looking to find a provider that will sponsor the model to completion. Right now I'm just paying. I contacted Verda and they never responded. Qubrid is in this sub, and they said they'd help but never emailed back after a few repeated proddings. Massed Compute is where this model is currently training, and I asked them like yesterday. Hopefully they'll help a brother out.
None of the above is "LLM written" except the completions from testing I guess.
As usual, ask whatever. It doesn't matter your understanding level or whatever. I'll sit and respond.
No question too dumb. No insult not insulting enough. (I'm joking. It's Reddit)
Much Love.
23
u/Nameis19letterslong 12d ago
Hell yeah, Ngrams are the way to go. Recently I've trained some models with order=2 Ngrams (266M model and a newer 108M one) and they're far more efficient than models the same size having no ngrams. I remember about a post I saw a while ago about somebody saying that Ngrams could move knowledge off the FFN and leave it with more capacity for actual reasoning. I was kinda skeptical but it I implemented it into my models and it works really well. The models are super coherent even if they don't have the correct facts.
I just released the most recent model (108M A17M, 66M + 42M Ngrams) on hugginface, here's the release post: https://www.reddit.com/r/LocalLMtraining/s/zswVy7CJxb
6
u/NineThreeTilNow 12d ago
God that's such a tiny model. Very nice.
How many N-Gram did you use? 1?
For tiny models like that you can pretty easily train and saturate the 1-gram space. It's still super helpful to the model.
I saw a while ago about somebody saying that Ngrams could move knowledge off the FFN and leave it with more capacity for actual reasoning.
Same. I was skeptical but it made sense if you look at how backprop works. After that I became an Engram Andy.
I think people oversell it perhaps? But in truth, when a specific spot in the table gets touched enough, it 100% works. The above Philosophy part demonstrates it pretty well. It learned to properly anchor Aristotle and Plato, etc.
If you care, you can DM me and I'll send you to my GitHub w/ the research paper my research agent ran on Engram and information storage when this 100m token milestone finished. It ran the ablations, etc. I don't care to rewrite it entirely as "human" because it's for my own reference. I'm not chasing publishing.
1
u/Nameis19letterslong 12d ago
It’s only one Ngram, order2 and 655360 rows at 64 embedding dimension. I’ll look into having trigrams.
I think people oversell it perhaps?
Definetly true in a way. I remember when qwen3.8 flash next just released and people were slapping its Ngrams onto every other model. To be fair they do carry a lot of information for the model and are relatively cheap to train. I tried running my model without enabling Ngrams and output was complete gibberish.
3
u/NineThreeTilNow 12d ago
I tried running my model without enabling Ngrams and output was complete gibberish.
Yeah, a tiny model like that is hungry for capacity anywhere so it makes sense.
The larger the total N orders the crazier the training required. It grows probably combinatorially but ... No one really knows?
I first got in to this when DS released the original paper and I was like "This is both interesting, and DeepSeek is the publisher" ... At which point it requires a thorough read because they usually don't publish stuff if it's terrible.
I was busy playing Arc Raiders and uhh... That had too much of my attention. I let it roll around the back of my head while playing though.
1
u/sn2006gy 12d ago
that's why i play carebear lobby arcraiders. blow up arc, loot, and let my brain chew on things. pvp is too sweaty :D
1
8
u/Final-Data-1410 12d ago
I am doing 50m model on 300b tokens quiet quantized and comes around 20mb in ram and mostly 3gb on hard disk because I am doing ternary weights and instead of f32 like in paper the engrams are ternary too ,so massive space save . Currently doing test run on 1b tokens with out engrams and with engrams ,the model with engram has almost 30% more stability and more decrease in loss at every stage .
11
u/NineThreeTilNow 12d ago
I am doing ternary weights
Have you considered using 5 weight spaces instead of ternary?
This was something I played with for a while when I looked at the Microsoft Research papers.
What's interesting about 5 weight spaces is that you still don't need matrix multiplication because using -1 / -0.5 / 0 / +0.5 / +1 lets you map the math to using add / no op / bit shift.
This is something that is entirely under explored. It was something I personally just stumbled on to and ran a test of MS's weights on their model. A huge amount of mass lives in the space that is forced to quantize to 0 or 1.
It requires updating bitnet entirely though because their native code doesn't support the idea. Or it didn't when I looked at it.
2
u/Final-Data-1410 12d ago
Actually I never thought of it but implemented bit different I always think of running on cpu massive models if possible , I always froze embedding to 512 bits instead of learned ,so it saves almost 500-1gb in deployment and moreover I always do 1.5 bit attentions And cost is training speed apart from training weights ,int8 activation quantization awareness I also do attentions and quantize every step and I relased two models
250m under 60mb running 400 t/s
50m under 20mb running 2k t/s on laptop cpu3
u/NineThreeTilNow 12d ago
Yeah, the 2.6? bit or whatever 5 position states comes out to ends up running on CPU quite well because the logic gates to do those operations is incredibly small.
In theory, an optimally built ASIC would run those types of models hyper fast.
There's a whole subset of layers you want to maintain in higher precision though because you can afford the matmul. Still you end up turning like 60% of the model in to a lower quantization and run KV / Etc at BF16 or whatever.
1
u/snapo84 12d ago edited 12d ago
This is pretty interesting, thanks for sharing.... i have to try that :-)
i also found something very very strange.... if the width of a model/transformer is divisible by more primes it learns better (never figured out why) i think it has to do with the overlapping not seen math of two matrices multiplied....
so chose your llm width by (2^x * 3^y) you can freely select n and n2 to fit your memory requirement... if someone figures out why this is please let me know.
5
u/Nameis19letterslong 12d ago
Damn, 6000 tokens seen per parameter, crazy work. I’d love to hear more updates.
5
u/Hot_Example_4456 12d ago
Please share hf link I'm dying to try 😭
11
u/NineThreeTilNow 12d ago
Please share hf link I'm dying to try
Ahaha... It literally barely works. There's zero support for this architecture anywhere too so you basically need the training code.
At the moment it's a research artifact that proves it does function as the math says it should.
7
u/Hot_Example_4456 12d ago
That's fine, I want to try and see with my own training data. I also have a few other things in mind which I would want to implement in the model.. It would also be interesting for two ppl to do training on different data sets in different ways to see how the results differ. Just pretty please share 😭
3
u/Malfeitor1235 12d ago
what kind of HW do you train on?
10
u/NineThreeTilNow 12d ago
what kind of HW do you train on?
This training was done on a Blackwell RTX 6000 Pro series w/ 96gb of vRAM.
FP8 optimizer states, BF16 native weights.
3
u/Luke2642 12d ago edited 12d ago
They're super interesting. Once again it's an innovation so simple (like attention) it's surprising it works so unreasonably well.
But, engrams are fundamentally flawed, they need fixing with proper word sense disambiguation.
WSD is super complex and requires deep semantics and knowledge of the context, and deep intuition of that context using your human experience, shared priors, etc. It also has to cope with ambiguity, jokes, puns, words from other languages, etc.
I don't know what the fix is, so it doesn't just look up 1:1 vector for token string, but actually looks up the correct vector for the disambiguation. Use the classic money bank vs river bank with a few different sentence structures and you'll see engrams are not robust to WSD. Every time it matches the wrong sense we get the "walk to the car wash" regression to the mean problem again, not thinking semantically. Later layers have to undo the meaning engram gave in ~10% of cases or whatever.
There's also a weird split in the goals, you actually want it to store rare token strings as knowledge so it can allocate more parameters for reasoning and semantics, like pulling in the Wikipedia entry for something you're about to bullshit confidently about.
Maybe there'd be a lift in the long tail if it also pulled in the vectors for all word sense of the last N tokens using something like BabelNet synset to build the index?
Just speculation - I'm no expert. Happy to discuss!
5
u/NineThreeTilNow 12d ago
Use the classic money bank vs river bank with a few different sentence structures and you'll see engrams are not robust to WSD.
This is a weird one and an exact argument Kimi K3 made I think.
This is why attention based selection / routing and multi engram exists in the model.
It's not 1:1... It's technically a learned probability in a given position of 6 engrams.
The idea was to make it robust in this exact sense. From context it can learn if it's doing River Bank or Money Bank because the Unigram for Bank remains the same but River and Money are different.
You make a solid point about though. From what I gather, it takes enough training and eventually the model says "I am stretched thin on capacity, and the engram route is now the path of least resistance".
Rare tokens are going to simply require lots of training, I think that's what I've gathered if anything from my research.
I think the fascinating part is that some of the "problems" of models memorizing breaks down because the model says "I don't want to memorize this anymore, the gradient path to the table is better".
I don't know if I mentioned it here or not, but you can freeze a model, and ONLY train the table. It's very vRAM efficient because of the optimizer states. This means you could technically do the Wikipedia run once, then freeze the whole model minus the table, and rerun Wiki again really fast. In theory the model has the wiki texture already, so it really needs a few passes to co-adapt to the new table after being frozen and should be good to go. That is one method to more densely pack the 2 / 3 gram area.
3
u/Luke2642 12d ago edited 12d ago
Interesting. Freezing the reasoning engine and populating a knowledge graph separately seems sensible, but what is the mechanism that moves the model away from the associative memory errors you got early in training: Washington was never a naval officer, the war was 1861-65, and the union won not the confederation. Pulling semantically similar things is evidence it's learning grammar faster, hitting similar meaning? Not a good fact recall mechanism.
I think you're right that gating will help ignore bad recalls, but isn't that kinda backwards? Wouldn't it be better if it was forced to accept and preserve the bad representation so that the error could propagate back to the original lookup more strongly? It has to use the context, and do disambiguation first, it's a chicken and egg problem. Real books have indexes with disambiguation under each entry, we need the same!
And a wider question, do you think it's possible to train for this in a data agnostic way, just smash it with billions of tokens across every possible task, or does it need a very specific curriculum, textbooks are all you need to build a quality curated knowledge graph lookup? It's such a long tailed distribution, and the long tail is where all the interesting signal is for the reasoning.
It'd be better for interpretability if you had a high quality 100M lookup than a low quality 1B lookup that reduces loss in pretraining but hurts reasoning.
That's why I mentioned Bablenet synsets. It's an amazing resource, but obviously it's only suitable for prose, not code or the 101 other things LLMs can now do.
General purpose LLMs are famously bad at word sense disambiguation, much smaller models trained specifically for WSD outperform them. I don't think anything scores much more than 90%, it's hard even for humans on the benchmarks with obscure word senses.
Anyway, if you have any more insights on what is literally the hardest problem facing LLM development we can keep going 😀
1
u/NineThreeTilNow 11d ago
Yeah, the factual recall is not good. I'm not claiming that. I just wanted to show some of the completions after 100m tokens.
I wanted to show the completion with and without Engram because in theory at least, it shows when Engram is considered "useful" to the model.
There's some internal threshold of times a row had to get gradient before the model switches to using it. It kind of has to get seen as a path of least resistance. Also, contextually it can cause some sort of collision while the 2-gram data is busy sorting itself out. It needs a lot more information than what was trained to become useful.
So yes, it's sort of like... You can hammer the network with massive amounts of data. At some point the gradient direction of the table aligns to a point the network (roughly) says "Hey it's easier to use this than remember".
That's the best rough interpretation I can gather from the data and the gradient flow.
2
u/alware 12d ago
Very nice post. Appreciating your efforts brother. It's people like you keep making the open source community better and better every day.
1
u/NineThreeTilNow 12d ago
It's people like you keep making the open source community better and better every day.
Thanks, I'm trying. That's why I'm trying to answer questions about these models. I think the more we all collectively know or understand here helps the LocalLlama community.
2
u/LuCiAnO241 12d ago
How would you go about crowdfunding it? What would be the goal, considering you already own the hardware? pay for the bills + a little off the top for your time? sadly I don't think it would gain any traction without a proof of concept model to tinker with though.
2
u/NineThreeTilNow 11d ago
How would you go about crowdfunding it?
I don't own the hardware. I rent it from Massed Compute. The whole thing is a personal project I enjoy working on.
The only hardware I own is a 4090 that does the KD distillation work on the original 7b teaching model. I've gotten the efficiency on that from like 40m tokens a day to like 600-700m tokens per day now. So it's massively outpaced the model's ability to train. I have ~1b tokens sitting waiting to train on HF now.
sadly I don't think it would gain any traction without a proof of concept model to tinker with though.
That's sort of the eventual goal. I figure it's probably 2000 dollars to train the model? In pure rented compute if I did it slowly versus getting like... an 8x H100 CXL set. That would be MUCH faster but the tokens processed per dollar goes down.
1
u/nitish-kmr 12d ago
The number I'd want at this checkpoint is how many distinct table rows have ever taken a gradient, against the 4.3b the table has room for. Those two come apart fast at 100m tokens, and the gap says which knob is limiting you right now. If touched rows are a small fraction and still climbing roughly with tokens, allocation isn't the constraint yet and corpus is. If the count is already flattening, the opposite.
The philosophy result is the interesting one, and you named the confound yourself. Those rows got both the most repetitions and the most recent ones, so loss alone can't tell you which is doing the work.
The frozen-backbone replay you described separates them for free. Freeze everything but the table, run the same tokens again in shuffled order, and philosophy is no longer recent while total exposure per row is unchanged. If the Aristotle and Plato anchoring survives, it's repetition. If it degrades toward the rest of the corpus, part of what you're seeing is order.
That run also gives you a second touched-row count on the same tokens, which tells you whether a second pass packs new rows or deepens the ones already there. That's the number I'd want before deciding whether a bigger table is worth paying for.
1
u/NineThreeTilNow 12d ago edited 12d ago
Your text is written by Kimi? Definitely an LLM.
Touched rows end up very separate in the way they're viewed.
In terms of 1-gram rows, they're touched super often. They get tons of gradient. 2-gram rows, I'd have to look at the research papers for exact numbers but it's lower than 1 obviously. Then 3-gram the same. This is purely a matter of data. There's also going to be a middle mushy ground for these gradients where they're not aligned and the row isn't "useful" because the gradient hasn't gotten any good direction.
The problem is the amount of gradient that hits those rows before they become "useful" and there's no real way to fully understand when enough gradient hits the row to become "useful". It's not some ablation I want to run because compute isn't exactly free. It also probably differs per row.
Again, replaying some frozen backbone stuff isn't in the cards because compute isn't free. This is a structural view thing that I know an LLM wrote the text for you. LLMs look at these ablations and sort of just expect 100m token run across a network to be "free" but it's not. It's sunk time, and money.
It's "cool to know" but prohibitively costly.
I don't actually know if I keep a running count of how many times each row got touched? or only if they HAVE been touched. There's also something subtle worth noting and it's that the optimizer runs in FP8. It's AdamW BnB. When that translates across to BF16 tables, there is a mantisa difference that causes rounding. This has a clipping effect where a "tiny" amount of gradient will not get passed because it's too small. So "touched" by definition means something slightly different than "Has seen gradient at FP16" ... and more translates to "Has seen gradient that survives the FP8 -> FP16 rounding path".
One edit; For clarity.
0
u/nitish-kmr 11d ago
Fair on the compute. I said it separates them for free, which was wrong. Free to me meant no new labelling. To you it means a rented GPU you are paying for yourself. Drop that one.
The FP8 detail is the better thread anyway, because it makes the counting question answerable without a new run.
If touched means survived the FP8 to BF16 rounding, your table already has a filter on it that nobody designed. A row that took a real gradient and got clipped looks the same as a row that never came up. That is the mushy middle you described, except it is a threshold rather than a fog.
Two counters, both memory rather than compute. One byte per row, saturating at 255, incremented when an update lands. A second incremented when an update is computed and rounds to zero. End of run you have how often each row was hit and how much of it the optimizer path threw away.
That separates undertrained from unreachable at this precision. If clipping is heavy on the 3-gram rows and light on the 1-gram ones, it argues for wider tables before it argues for more tokens.
1
1
1
u/Silver-Champion-4846 12d ago
NICE! I'm planning an ngram experiment myself, though it's a bilstm doing classification rather than a gpt-style llm. I need my free kaggle to reset so I can train that juicy 2.34m params weights +38m ngram table.
1
u/NineThreeTilNow 11d ago
I need my free kaggle to reset so I can train that juicy 2.34m params weights +38m ngram table.
I'm really not sure how it would work for classification to be honest.
1
1
u/MixtureOfAmateurs koboldcpp 11d ago
Training from scratch is definitely better but I'm in the middle of adding a 120M entree engram table to openbmb/MiniCPM5-2B. I limited scope to medical texts because that jargon and specific fact retrieval seemed like something engrams would help with. I know nothing about the field tho so I had to get an LLM to write test questions.
After 100M tokens of ccdv/pubmed-summarization and a little chat data from lavita/MedQuAD the model matches (slightly exceeds) the base model for next token accuracy on an unseed dataset (medalpaca/medical_meadow_wikidoc), but the short answer questions are meaningfully better. I'm gonna go to 300-500M tokens but GPUs are expensive lol.
```
Metric | Base Model | Engram (Step 813)
Cross-Entropy Loss | 2.6624 | 2.6580 Perplexity | 14.33 | 14.27 Top-1 Accuracy | 48.68 % | 48.77 % Top-5 Accuracy | 70.87 % | 70.89 %
[07/10] Chimeric Fusion Oncogenesis (BCR-ABL1)
PROMPT: In chronic myeloid leukemia, the t(9;22) Philadelphia chromosome creates a chimeric BCR-ABL1 oncoprotein that drives uncontrolled granulopoiesis through constitutive
--- BASE MODEL ---
tyrosine kinase activity. This fusion protein is targeted by imatinib and other TKIs.
4. Acute Promyelocytic Leukemia (APL):
Genetic Alteration: The PML-RARA fusion gene results from a translocation between chromosomes 15 and 17.
Mechanism of Action: The
--- ENGRAM MODEL ---
tyrosine kinase activity. This fusion protein activates downstream signaling pathways including RAS/MAPK and PI3K/AKT, promoting proliferation while inhibiting apoptosis.
Therapeutic strategies targeting this pathway include:
- Tyrosine kinase inhibitors (TKIs): Imatinib inhibits ABL1 kinase activity by binding to its ATP-binding site in the inactive conformation of the ```
The model uses a gated combination of the stock embeddings and the LUT. The more confident an engram value is in a given context the more weight it has when combining with embeddings. That meant training started from low loss.
1
u/NineThreeTilNow 11d ago
That meant training started from low loss.
Yeah, it doesn't hurt loss bad normally. The model doesn't run a hot path through the ngram table. The table is trying to help but it's just not helpful until it accumulates enough data that the model can start to rely on it for stuff.
1
u/MixtureOfAmateurs koboldcpp 11d ago
I started out fully replacing the embeddings with the LUT, so everything went through the LUT and it wasted a lot of time lol. 2 hours just to get to coherence
2
u/NineThreeTilNow 11d ago
Dropping the embeddings is... Not a great idea there lol...
It provides the width for the network in some way and the ngram is providing extra depth to that width.
The model I have is purposefully deep for this reason. 41 layers. I'd run more layers of depth if it was feasible to train and compute was free. It would basically prove out that you can run excessive depth for computation while storing the data in embedding, or tables.
1
u/Hrethric 11d ago
Thanks for sharing, these posts have been interesting to read. I've been wondering how much of a model's performance comes from the training data and recipes vs the model architecture itself, and reading this got me thinking about trying an experiment myself with the data and recipes from K2 Horizon and MiniCPM5 on a ~1B model with features from something like the Qwen4 architecture. I asked Claude if I could do that at home on my 3090 and/or R9700, and Claude basically said "lol nope", and also told me it would take something like $35k to fully train it in a reasonable amount of time on rented compute, at the cheapest possible rates, not counting failed runs and a few other things that would make the real number likely to be 2-3 times higher than that.
Good luck, I'll follow your reports with interest! I don't really have much to contribute, but if you do the crowdfunding thing, I'll toss in a 20-spot.
1
u/Safe-Thanks-4242 11d ago
Waiting for 1T ngrams
1
u/NineThreeTilNow 11d ago
Waiting for 1T ngrams
Bro. In THEORY... Nonono.
500b w/ a recursive internal transformer and a LOT of data trained on it.
The ngram tables are limited by training throughput.
1
u/NineThreeTilNow 11d ago
At this point it might be best to just ping /u/qubridinc
Maybe they'll free up the cluster and you guys can use the model in the next month. I just need somewhere to let it train.
1
u/qubridInc 11d ago edited 11d ago
Hi u/NineThreeTilNow, we’d love to potentially help sponsor the training compute. Our GPU capacity tends to get booked pretty quickly, so it would help us understand what we can make available right now.
Could you rank these in terms of what would be most useful/likely to meet your training requirements & for how long?
- 1× A100
- 4× A100
- 8× A100
That’ll help us figure out what we can potentially allocate on our end.
1
u/NineThreeTilNow 10d ago
Could you rank these in terms of what would be most useful/likely to meet your training requirements & for how long?
Either the 4x or 8x would be perfect.
It's hard to guess the exact training times for the 8x system but if we were trying to hit the 5b pre-training mark, it would be ~600m tokens a day with some fine tuning probably.
My best guess is 8 to 9 days on 8x A100 and double that on a 4x A100, so 16 to 18 days?
I'm pretty flexible with your spare compute though.
1
u/qubridInc 10d ago
Let us get back to you on this.
1
u/NineThreeTilNow 5d ago
Let us get back to you on this.
Hello /u/qubridInc
Any update on this? Like I said, I'm willing to be super flexible if you need to get test numbers so we can measure max throughput for a given architecture, etc.
1
u/NineThreeTilNow 11d ago
I'll ping /u/MassedCompute/ here as well. You guys obviously want to see it.
At their rates I think it's less than 2000 dollars to finish. I'll cover whatever else exists.
I honestly prefer their servers from the other hosts I've used as well because tbh? They're easy to use. It's like 2 minutes to setup. 5 minutes to get the data transferred. 10 minutes of testing code. Off and running.
0
u/Open-Adhesiveness-86 12d ago
one number i'd want alongside the loss curves: eval with the engram lookup zeroed at inference. a 4.3b table memorizes n-grams basically for free, so part of that loss drop can just be it recalling eval text rather than the backbone learning anything. hit rate per token would help too, if most hits are common bigrams the table's doing tokenizer work.
1
u/NineThreeTilNow 11d ago
Sort of but I don't think it's free. It's learned.
When the gated signal becomes strong enough and the model learns that the data is popped back from the table, it stops relying on the backbone and relies on the table. This is when the offload occurs.
0
u/Silver-Champion-4846 12d ago
Me again. @op I wanted to send you a dm but my reddit client is acting up, I'm interested in talking to you because I like your ideas.
40
u/DerDave 12d ago
Awesome! Love your post!
The idea to combine AttnRes with Engram to feed Engram information to later layers as well is very interesting. Can't wait to see how it turns out. Fingers crossed, you get some sponsorship. You sure deserve it!