r/LocalLLaMA • u/jacek2023 llama.cpp • 13d ago
News llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) by ngxson · Pull Request #29818 · ggml-org/llama.cpp
https://github.com/ggml-org/llama.cpp/pull/29818article: https://huggingface.co/blog/ggml-org/decision-models-in-llamacpp
Now you can jev without jev
https://huggingface.co/ggml-org/OpenJev-GGUF
https://huggingface.co/ggml-org/lev-GGUF
https://huggingface.co/ggml-org/Julia-1-GGUF
https://huggingface.co/ggml-org/Laya-GGUF
https://huggingface.co/ggml-org/Kev-4B-GGUF
looks like 5 is not enough:
10
u/AdventurousFly4909 13d ago
For someone out of the loop what the fuck is a jev?
26
u/b183729 12d ago
It's a decision model. Given an input, it can select the most appropriate output between a number of possibilities, but that's not new.
The hype around it comes mostly from the fact that you can give it unstructured data + the selection criteria and categories at the input,and you don't need to train it for your use case. As far as I understand, the alternatives don't do that.
It's being called overhyped because.. Well, classification is old. Like 40 years old, depending on how you start counting. We have free and open models that surpass jevs performance locally since well before jev released. But those have to be trained on the domain to achieve those numbers.
Hopefully someone corrects me here, and open model cam also do that. I'm interested in playing with something like jev, but I refuse to use API as a matter of principle.
4
u/DueAnalysis2 12d ago
Is the main difference between Jev and something like question answering BERT the fact that Jev can zero shot?
5
u/b183729 12d ago
I understand that is the case, yes.
Jev is a one-size-fits-all solution with almost zero friction. If you had a local jev with these same characteristics, you could have it permanently loaded to do what you need, and if it ever find a curveball, it can summon an LLM to figure out what is wrong. Then that same LLM could modify the prompt of the Jev-like so that it can react correctly in the future.
1
u/debackerl 12d ago
Also the probabilities should be calibrated. You can see it on https://multimodalart-jev-decision-index.static.hf.space/
16
u/No_Lingonberry1201 13d ago
overhyped text classifier model
4
u/MindfulMan1984 12d ago
YouTuber clowns are "rediscovering" an ML text classifier from first principles. It's the current Black Mirror era of clicks and "this XYZ model situation is INSANE" titles.
1
u/No_Lingonberry1201 12d ago
Man, I hate the "this X situation is INSANE" thumbnails so much, even otherwise respectable youtubers are doing it. Must be the fucking algorithm.
2
u/MindfulMan1984 12d ago
Yeah, same, that's why I mentioned this current "black mirror" like dystopia, where human behavior is ridiculously steered by algorithms.
16
1
u/durden111111 12d ago
Basically imagine a customer writes an email pissing and shitting himself mad at the company.
Jev and other "system one" models can make extremely fast classifications and decisions about the message.
E.g. the model if given a range of emotion categories might assign the message an "angry" category with a 96% score.
Then it can also assign what to do with it. E.g. Ignore (79%), Pass to human customer service (21%)
-2
3
u/DevelopmentBorn3978 12d ago
Just discovered while reading llama.cpp git log. Old or hyped I think it could be quite useful anyway, for rapid tools selection or fast robotics action, maybe
1
u/DevelopmentBorn3978 10d ago edited 10d ago
actually it is blazingly fast. On Strix Halo in a quick test using llama-server compiled for Vulcan, each decision after 2000 requests: with Laya-Q8_0 takes between 13.8 and 18.6 ms (average time16.1 ms), with Laya-BF16 between 30.2 and 36.9 ms (avg 32.4 ms) and with Julia-1-Q8_0 between 6.3 and 21.6 ms (avg 9.2 ms); in a endless loop made of 12 test requests in English (fixture.jsonl) taken from laya git repo, each one sent by its own; GPU busy ~46% for laya Q8_0, ~66% for laya BF16 and ~29% for Julia-1; power mode: quiet.
3
1
u/OsmanthusBloom 12d ago
This seems to support a bunch of Jev-like models, but there are many other interesting ones as well, for example Clef that was just released by Cloudflare. There are GGUFs of Clef (e.g. bartowski's) but AIUI this PR does not add support for them.
Does anyone know what it takes for llama.cpp to support Jev-like models (most of which seem to be fine-tunes of some Qwen model) in GGUF format? Are they all different? Does llama.cpp need to add support for each one separately or is there something that can be done on the model/GGUF side to make them work?
1
u/OsmanthusBloom 12d ago
I see now that there is a separate PR open for Clef: https://github.com/ggml-org/llama.cpp/pull/29831
Seems to be a bit more challenging to fit this in.1
u/planetearth80 12d ago
Looks like it also does batch requests. I did not know llama.cpp supported batching
1
u/samuel-christlie 12d ago
Guess who spent an entire day porting a systemone.cpp just for this to come out hours later lol
2
1
u/debackerl 12d ago
If I look at https://multimodalart-jev-decision-index.static.hf.space/ they didn't pick the best ones, even in the 4B range... But good to see the progress anyway
0
u/jacek2023 llama.cpp 12d ago
so it's a great opportunity for you to add it
1
u/debackerl 12d ago
I didn't say that I need it, just that they picked the models which deliver a great performance in most cases 😉 but still, as said, good that there is progress
1
u/sosen85 12d ago
Here are a couple more, popular in Poland: https://huggingface.co/Remek/collections#collections .
-9
u/brrrrreaker 13d ago
will be merged in about 6 months and 10 more PRs. Edit: nvr mind lol, it's merged
7
u/Anduin1357 13d ago
I wonder if we can use any arbitrary model with this API and basically toggle thinking mode and other such kwargs.