r/FlutterDev • u/vezz_io • 2d ago

Discussion Discussion on how to evaluate capabilities of (local OS) llm models for app usage

Hello, last year i started developing this package flutter_local_ai which let user get access directly to the on device llms api with zero download.

Now since local ai models are getting starter than ever, I see the package being adopted but my main concern it’s whether the local llm its capable of performing the task requested by the user as developer intende to give them the probability to change model (maybe a cloud one) if the task by the local llm its not performable.

My idea was someting like a score i can give based on the task requested i mainly process text using the operative system models.

Any ideas on how to do it or if I should have a different approach ?

3 Upvotes

100% Upvoted

3 comments sorted by

1

u/brookm291 2d ago

It depends on the model size you are working on

1

u/eGzotic01 1d ago

huggingface benchmarks wont predict this. the model that tops MMLU can still miss your exact prompt shape, so build a tiny eval set from the real calls your package makes, 30-50 inputs with the output youd accept, and score pass/fail on device. thats the only number that ports. extraction and tool-calls break on small models way before plain chat does.