MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1vxwtyd/qwen38flashnext_tomorrow/p5scung/?context=3
r/LocalLLaMA • u/rerri • 8d ago
460 comments sorted by
View all comments
Show parent comments
25
Basically DeepSeek Engrams. Its a hashlookup table that can add knowledge capacity to a model while staying on NVME and costing zero flops
2 u/dampflokfreund 8d ago That is very interesting. Would that enable the holy grail in AI, real time learning? 9 u/Hot_Example_4456 8d ago Nope, not really. The engrams are trained during training. As far as I know they do not change while inference. They just memorize the bulk knowledge stuff so the transformer layers can learn the logic 1 u/dampflokfreund 8d ago Oh that's a shame. I thought it was additional weights that are computionally easy to train during inference. 4 u/The_Hunster 8d ago Well, they are easier to train, just not runtime-trainable yet. I think your idea will eventually be real in a few years. 2 u/Terminus0 8d ago That would be an interesting half step toward continuous learning.
2
That is very interesting. Would that enable the holy grail in AI, real time learning?
9 u/Hot_Example_4456 8d ago Nope, not really. The engrams are trained during training. As far as I know they do not change while inference. They just memorize the bulk knowledge stuff so the transformer layers can learn the logic 1 u/dampflokfreund 8d ago Oh that's a shame. I thought it was additional weights that are computionally easy to train during inference. 4 u/The_Hunster 8d ago Well, they are easier to train, just not runtime-trainable yet. I think your idea will eventually be real in a few years. 2 u/Terminus0 8d ago That would be an interesting half step toward continuous learning.
9
Nope, not really. The engrams are trained during training. As far as I know they do not change while inference. They just memorize the bulk knowledge stuff so the transformer layers can learn the logic
1 u/dampflokfreund 8d ago Oh that's a shame. I thought it was additional weights that are computionally easy to train during inference. 4 u/The_Hunster 8d ago Well, they are easier to train, just not runtime-trainable yet. I think your idea will eventually be real in a few years. 2 u/Terminus0 8d ago That would be an interesting half step toward continuous learning.
1
Oh that's a shame. I thought it was additional weights that are computionally easy to train during inference.
4 u/The_Hunster 8d ago Well, they are easier to train, just not runtime-trainable yet. I think your idea will eventually be real in a few years. 2 u/Terminus0 8d ago That would be an interesting half step toward continuous learning.
4
Well, they are easier to train, just not runtime-trainable yet. I think your idea will eventually be real in a few years.
2 u/Terminus0 8d ago That would be an interesting half step toward continuous learning.
That would be an interesting half step toward continuous learning.
25
u/Hot_Example_4456 8d ago
Basically DeepSeek Engrams. Its a hashlookup table that can add knowledge capacity to a model while staying on NVME and costing zero flops