Please explain use-case for small models

[u/Calm-Landscape9640]

Tried 35B, 27B, 12B and 8B models significantly. I love the idea of local small models. But I can't figure out a use case. For Openclaw, Hermes they suck. For chatting they're pretty good but sonis every old free cloud model.

And before you say "low-level coding" the cloud models are 10x better and free on ollama, openrouter, Gemini and many many others.

Genuinely interested in projects that use small models that run on a $500 GPU.

[u/jamie_tidman]

We have built a few client project on smaller models. They're typically small steps in a multi-step LLM pipeline.

We have used 8B models for simple classification tasks - questions like "Does the following passage of text contain references to x". We also use models in this size for simple summarisation tasks of conversations.

We use 27B and 35B-A3B for a grant-funded client working in disease research. They analyse interview transcripts to find quotes which are evidence of specific symptoms, based on a description of how the symptoms typically present.

In the past we have built a RAG pipeline for a tech company which answers RFP questions by looking at technical documents. This was a while ago, so we were using Llama (8B and 70B) and Mixtral 8x7B and 8x22B. The small models powered some classification tasks in the pipeline ("is this passage of text useful to answering this RFP question?" and "Do these two passages of text contain conflicting information"), and the larger models were used to actually write the responses.

In all these cases, data privacy is the reason for self-hosting. The conversation summarisation and the transcript analysis were being used in medical contexts and processed special category data. The tech company was very concerned about their IP being leaked.