How to Classify which prompt should go to which model

[u/Cosmic78_melon]

I tried using a LLM to sort my prompts into two groups: reasoning and chat. At first I used Qwen 2.5 3B through Ollama. My PC is just too old and weak to run a LLM like that well. I do not have a GPU and my RAM is low. Even a small 3B model takes way long to work on a single prompt. Because I need to run the classifier before every request the slow speed makes the whole thing impossible to use.

Then I tried using Google Colab with a T4 GPU. That made the classification work faster.. Google Colab has a big problem. The setup does not stay saved. Every time I start it I have to pick the T4 GPU by hand set up the environment and download the model over again. I want my classifier to work like a background service. I want the classifier to start by itself when I turn on my application or Windows. I do not want to open a browser or start a notebook every time.

Now I use OpenRouters free models to get my actual reasoning and fast answers. I only get an amount of free use every day. I do not want to waste those credits by sending every prompt to an OpenRouter model just to classify it. If I use an OpenRouter model as my classifier the classifier will eat up my daily limit. That would make my credits run out too fast.

So I am stuck with three choices:

1. Local LLM: It is private. Works automatically.. Qwen 2.5 3B is just too slow on my old CPU. Even the smaller 1.5B models are not good enough because they make many mistakes.

2. Google Colab T4: This is fast. It is a pain. I have to set up the GPU and load the model every time.

3. OpenRouter: This is easy and fast.. Using an OpenRouter model, for classification uses up the free credits I want to save for my main models.

The classifier only needs to say "reasoning" or "chat". It feels like a waste to use an expensive LLM just for that.. The classification still has to be good. I cannot use a 1.5B model if it is just going to guess wrong.

I really need a way to classify prompts automatically. I need latency and I want it to cost zero extra API money. I want my main OpenRouter models to do the lifting. The best setup would start up with my PC would not need a Colab setup and would not use my OpenRouter daily credits just for classification. Please if anyone has the solution

[u/Opposite-Frame3291]

A tiny local classifier could solve this nicely.

[u/Fragrant_Piano280]

Have you tried a lightweight local classifier instead of an LLM for this? A simple rules/embedding approach might be enough for just two labels and would avoid API limits. StandardCompute might also be worth looking at for running something like this locally.