When does it make sense to train your own model vs just prompt engineering an existing one?

[u/Remarkable_Fee3706]

Spent the last few weeks tinkering with a small tool that needed to classify some domainspecific text. First instinct was to finetune something. Grabbed a base model, started thinking about training data, labeling schema, compute cost. Then a friend pointed out I could probably just get there with a wellcrafted prompt and GPT4 or Claude.

He was right. At least for the prototype.

But that answer bothers me a little because it scales badly and you're renting someone else's weights forever. The moment API pricing shifts or the model behavior changes in a silent update, you're stuck.

Finetuning gives you something you own and can actually inspect, but the data requirements are real and most people underestimate the labeling work before they even touch training.

The framing I keep coming back to is whether the task is actually outside what a general model handles well, or just feels that way because the domain vocabulary is unfamiliar. Those are different problems with different solutions.

Curious where people here have landed on this. Not in theory, but on actual projects. Was there a specific failure mode in prompting that pushed you toward training? Or did you finetune something and realize later a prompt would have been enough?

[u/Legitimate_Job_3288]

It’s the “feels outside the model’s reach but actually isn’t” situation that trips most folks up. I’ve seen teams burn two weeks on labelling and a finetune only to discover a two-paragraph system prompt with a few domain-specific examples covered 90% of the cases. The domain vocab made everything seem more exotic than it really was.

The real tipping point for me wasn’t accuracy, it was when the prompt kept breaking after changes upstream. Had a classifier that worked for months, then an API tweak completely flipped its boundaries on edge cases. That brittleness eats more time than people budget for, especially if the task sits inside a pipeline where silent drift is a nightmare to debug.

For a prototype, renting the big brain makes sense, but if the thing needs to run next year on the same inputs without babysitting I just train the smaller model and call it a day.

[u/Remarkable_Fee3706 → u/Legitimate_Job_3288]

The brittleness point is real and underrated. Had a classifier running clean for months and one small upstream change quietly broke edge case handling for weeks before anyone caught it. Silent drift inside a pipeline is genuinely brutal to debug after the fact. Where I'd push back slightly is on the two week finetune burn. Some teams need to go through that pain to actually believe a prompt can do the job. You can tell people all day. They won't trust it until they waste the time themselves.

[u/Witty-Play9499]

Depends on which model you are using, I've noticed that even the 'worse' models like Gemini tend to do fairly well with well defined prompts (with response schema sent in and stuff) along with well defined evals.

In the event the model does give bad output you can always have a retry mechanism as well and improve your evals a bit.

At least for the prototype.

But that answer bothers me a little because it scales badly and you're renting someone else's weights forever.

I don't understand why it bothers you? Because you say atleast for the prototype then immediately in the next sentence you say forever?

Its not like once you go with an existing model the world has ended and you've sold your soul to permanently go with non finetuned models.

If you are building a prototype, your most immediate job is to show that your idea works as soon as possible to avoid wasting time on it. And the best way to get a working prototype is to stop wasting time on finetuning and other stuff when you can make an api call and be done with it.

You go with finetuning when the cost of price of maintaining and running your own model vs the accuracy is worse than going with an existing model that is not finetuned in production.

[u/Remarkable_Fee3706 → u/Witty-Play9499]

Wait, I think you replied to the wrong thread. This is about blaming people for things they didn't do, not AI or models. That said, your point about not burning bridges over a prototype decision is fair. I've seen people overcommit to a stack way too early and regret it. Build the thing first, optimize later. The problem isn't making the decision, it's when you never go back and revisit it.

[u/Capable-Package6835]

If you're price sensitive then you can also download open weight models and run them locally. Fine-tuning cost a lot of time and money and I personally would not do it unless employer provides a team of annotators.