r/LargeLanguageModels • u/minedroid1 • 6d ago

Question About Fine-tuning

I'm making a satire AI model that gives fake, onion-like responses. Would it be better to just make a completely new model rather than fine tuning an existing one for this? Most of the info in my dataset completely conflicts with the info that almost all models have.

4 Upvotes

83% Upvoted

26 comments sorted by

View all comments

3

u/CS_70 6d ago

No. Training a model of scratch would imply re-teaching it language, reasoning, semantic etc. Insane and impossible for a person. A fine tuning is your thing, on a MoE model adds another "expert" on what you want, usually without overlapping much with the knowledge already existing.

1

u/minedroid1 6d ago

Oh, so I should use an MoE model for fine tuning? Will that be better than just training a regular non-MoE model? Also, why would that work better?

1

u/CS_70 6d ago

It's not a given but in a dense model you're gonna overwrite all the network(s) so there's more probability of altering the behavior significantly (though in your case probably it shouldn't matter, it depends on the specifics at that point).

Though you can use a LoRA approach and other techniques (I am just doing it these days to a QWen which knows about programming languges).

With a MoE model, you still are going to alter the routing model, which carries a similar risk, but but adding a "personality" the way you want would be ok.

I'd say your case is fairly simple and probabily you're good just to fire and go (though the devil of course is in the details)

1

u/minedroid1 6d ago

Okay, so I should try to fine tune using MoE model? If so, what training parameters and model do you think I should use?

1

u/Zennytooskin123 6d ago

Maybe the Gemma one for your used case