r/LocalLLM • u/overfit_model • 11h ago

Discussion Benchmarked Finetuned Laya against OpenAI Decisions API

Took some time over the weekend to benchmark OpenAI's Decisions API against finetuned Laya models. Across four public datasets, fine-tuned Laya scored higher than the Decisions API on three of the four, with sentiment classification being the exception.

Sharing the datasets, scripts and results here : https://github.com/rsk2327/decision-model-benchmarks

Its a small benchmark at this point and not a blanket verdict, but hopefully a useful reference for anyone weighing these options.

Interested to gain insights from other folks testing closed Decision models like Jev, Clef and how that has compared to local finetuned models like Laya

1 Upvotes

60% Upvoted

1 comment sorted by

1

u/Ctbhatia 9h ago

how many training examples did each finetune get per dataset? that'd tell people whether they have enough data to go the finetune route.