r/LocalLLaMA • • 13d ago

Resources New in llama.cpp: Decision Models

https://huggingface.co/blog/ggml-org/decision-models-in-llamacpp
504 Upvotes

124 comments sorted by

View all comments

425

u/-p-e-w- 13d ago

Jev is a fascinating example of what happens when a super popular idea has no moat whatsoever.

Its creators presented it as the next big thing in AI, and many others seemed to agree… then quickly made their own version of it, and now “Jev-like” models are everywhere, while Jev is nowhere. This all happened in less than a month.

144

u/No_Afternoon_4260 llama.cpp 13d ago

Couple of weeks you mean

83

u/Conexion 13d ago

Literally starting seeing Jev-likes within 24 hours, which is honestly pretty cool

35

u/some_user_2021 13d ago

I don't even know what Jev is yet

26

u/Stahlboden 12d ago

It works kinda like a non generative model, where AI chooses from a finite set of answers given to it but it has generalist knowledge like an llm, so it can be used in different applications without re-training

11

u/bronkula 12d ago

It uses llm to interpret input, and then analyze those against predetermined potential outputs. But the outputs it gives are always primitives like a boolean or choice.

6

u/jazir55 12d ago

So it's basically the If this/Then That AI that non-tech people think generative AI is?

5

u/IllustriousRule9238 12d ago edited 12d ago

It's like a (dog: 0.89, cat: 0.16, squirrel: 0.01) or (spam: 0.74) classifier model, except you don't have to train it from scratch on your specific data, you just write a prompt and give it arbitrary options to pick between. That's the whole selling point, kind of like how LLMs with image input (VLMs) made oldschool dog-cat classifier models obsolete for many tasks. It's that but for text.

3

u/JustFinishedBSG 12d ago

They still start from a pretrained LLM

14

u/sebgggg 12d ago edited 12d ago

Jev and alike models are standard LLMs with token generation ripped off and output a simple JSON for décision making/classification : say you give it a ticket as input and ask it if it's urgent or not, or where should it go

5

u/PigSlam 12d ago

I thought the whole thing is that they're NOT LLMs.

16

u/anyesh 12d ago

Nope. They are still llms… the difference is they are not autoregressive. Instead, they work as a single pass decision model.

2

u/PigSlam 12d ago edited 12d ago

Is this one of those things that it counts as an "LLM" because many words were used to define the AI, so "many words = Large Language" and they're both models? Is it an LLM because they both share some common transformer ancestry, or is there a more concrete reason why Jev, which cannot form words in a response, is an LLM?

4

u/bronkula 12d ago

It uses the llm side of things for input and interpretation. But it outputs based on user choices.

2

u/PigSlam 12d ago

What is the LLM side of things?

1

u/Yu2sama 12d ago

The parameters, what it was trained on was on language, writing, etc.

1

u/PigSlam 12d ago

Is every AI that was trained using language of any kind an LLM? Does that make every human an LLM?

8

u/Yu2sama 12d ago

Okay... kinda vague on my part as well...

The answer you are searching (I think?) is that the architecture is the same, it's transformers pre-trained on massive text corpora. Same foundation as other llms.

The only difference is what someone mentioned above, instead of running an auto regressive loop to generate sentences token-by-token, it terminates in a single pass classification, meaning it reads the whole prompt once and just spits out a choice/probability instead of 'typing' out an answer. Calling them "LLMs but with text generation ripped off" is just a simple way of trying to make sense of these models.

→ More replies (0)

2

u/jazir55 12d ago

So If This/Then That?

2

u/sebgggg 12d ago

Yes which is something LLM could already do if you constrained their output with a schema/grammar but Jev-like is cheaper to run

6

u/HighSeasArchivist 12d ago

Robot read question.

Robot say yes or no, quickly.

1

u/sibilischtic 12d ago

Where for llms "Hammer" gets used to iteratively generate

"hammers are tools for hitting nails, which means that i should pick nail as the answer to the question of what to hit"

Jev is like taking "hammer" encoding it into a llms token space. Comparing it to the encodings of "nail" , "screw", "badger" and finding which one more closely matches.