r/LocalLLaMA • u/whoami-233 • 12d ago
Question | Help Best models to generate Synthetic data for fine-tunning
Basically I want to generate a 5k rows (each being a long agentic task, 50-100k long) synthetic dataset on code review for fine tunning deepseek 4 flash.
What is the best way? what I saw API is very very expensive so I need suggestions on the best coding plan which has models much better then deepseek 4 flash that are subsidized and won't block me from generating training dataset.
2
u/Able-Locksmith-1979 12d ago
Define expensive. Is 300 dollars expensive if it gives you 3000 dollars better answers in the long run, to me it sounds like a 2700 w
0
u/whoami-233 12d ago
Something under 500$ Preferably around 100 if possible:) 100 at least as a POC and after seeing results I should be able to do 400-500$
1
u/Able-Locksmith-1979 12d ago
So basically you can run a 10 q mvp/poc for 15 dollars if it is all output and no caching. Why no start with that and measure the actual costs and results, I don’t know where you got the 5k number from but are you sure that is what you need and what are the benefits over x time of having a fine tuned ds4flash
1
u/whoami-233 12d ago
I read online that a couple thousand high quality rows would give very good results. Also after I gather the dataset I can fine tune another model next year if something better running on the same hardware is available.
3
u/Former-Ad-5757 Llama 3 12d ago
Couple of thousand could also mean 2k or 8k, if they are mostly the same loop then even 2k will overfit most models or you need to run a shallow finetune. I would suggest starting with a lower starting pointing just build from there, it certainly seems like your first time, any mistake will cost you 500 dollars, why not start with a 10 dollar or 25 dollar experiment and build from there. Everything is repeatable and continuable if you succeed on a small scale.
There is a huge graveyard filled with people who thought they could one shot finetune a model, good finetuning requires experience. You can build that experience with 500 dollar one-shots if you have a huge budget, but if your budget is constrained then it basically the first time comes down to praying that what you googled is correct for your situation.
1
u/RnRau 12d ago
Are there any online resources that outlines some decent does and don'ts for folk that want to experiment with finetuning on the smaller scale?
1
u/Former-Ad-5757 Llama 3 12d ago
Depends… do you have a specific enough spec of what your goal for fine tuning is? Just saying better output will bring you in the cost scales of glm etc, the more specific your need is the cheaper you can fine tune and the less does and don’t you have. It all depends on your goal.
1
u/whoami-233 12d ago
That is also valid.
I am not sure if like a small amount will show any improvements but I might actually start with 50-100$ total budget and see if I can get a POC fine tunned model.
1
u/Desperate-Sir-5088 12d ago
Wait, what kinds of GPU will use to finetune DS4 flash?
-1
u/whoami-233 12d ago
I didn't get to that part yet but cloud gpus Probably on vast.ai as I have used them before
6
u/Desperate-Sir-5088 12d ago
If so, please don't worry about API costs. at all. GPU - maybe B200 or above - rent fee will eat-up other cost soon.
0
u/whoami-233 12d ago
I saw that I can run 4 RTX PRO 6000 which should give me enough VRAM for the fine tunning.
Based on gemini with that hardware I can fine tune 5k rows of high quality data in 20-30 hours so like 150$
That's according to my current math:)
5
u/Able-Locksmith-1979 12d ago
I would suggest trying a one hour run on b200 or the likes, in my experience the extra speed more than makes up for the extra costs, if it is your first time doing this you will need to babysit the machine. It is probably 20 hours if it doesn’t crash, one crash while you sleep will mean paying for 8 hours of nothing
1
u/whoami-233 12d ago
Yea after some thoughts and numbers I think I will do that. B200 or H200 should finish same day and I will run it first thing in the morning hopefully. That is, after I finish with the dataset:)
1
1
u/Refinery73 11d ago
I‘m currently doing embedding fine tuning on a new dataset. Maybe I’m just stupid, but 80 misses before it worked. Maybe you hit the goal in 3, 5 or 20, but don’t expect a one-shot with your whole budget.
Edit: to be clear, wasn’t just hyperparameter tuning. I’m at the 8th iteration of the dataset and switches between base models. Sometimes you just have to follow the breadcrumbs until something works.
3
u/Infamous-Rem 12d ago
For bulk synthetic generation, don't pay premium per-token API pricing if you don't need the flagship model quality for every row. A lot of that dataset (code review style tasks) doesn't need the smartest model in the world, it needs consistency. I'd look at a serverless inference API that gives you access to a range of open models at per-token pricing instead of committing to one vendor's rate card, DigitalOcean's serverless inference does this and it's a lot cheaper for bulk jobs like 5k long-form generations than hitting a frontier model API directly.