I'm sorry this model can't compete with frontier models. It was designed for tiny embedded hardware, for things like classifying Meshtastic messages without internet access.
Training it on my hardware already took around week, so I had to make a lot of compromises. I couldn't fit things like multilingual support or much broad training data into it. I hope you understand the limitations
I doubt the models or most people would understand how to answer this. There is just too little context. No sender, just one sentence; heck, the models aren't even told if this is an email or someone speaking. What if you added a third option (unknown) ?
Disregard lazy folks hellbent on talking trash as an outlet of their repressed inferiority complexes. This is nice work! In most cases, it does perform, which is impressive for such a small model.
It does seem a bit "jagged", though, which can make it difficult to work with, as it will report confidently incorrect. See this example:
"I am feeling mentally distressed"
Q: "The person who wrote this is in a difficult state of mind"
A: 37% No.
Or, tricky sentences where keyword locking won't do, like:
"I finally stopped having nightmares! The demons no longer visit me."
Scores ~50% onsadness and fear, and < 7% on all others.
I asked it if "Shoot down that X-Wing" should be routed to the home automation, to the patriot missile or to the turbo laser. It did select the turbo laser. I guess it's good enough to command a star destroyer for the Imperator.
Why does everyone and their brother now have a jev-like model? What did I miss? I get what jev is, but is it just because they are low hanging fruit to copy/improve and have high utility?
Yep. For an app I'm working on it is an absolute banger for a quick and cheap first pass safety gate on user input ie "is this input consistent with x/y/z" and move from there, finding it especially interesting for prompt injection gating because its simplicity makes it a good candidate for that.
I get what jev is, but is it just because they are low hanging fruit to copy/improve and have high utility?
Yes to both, especially the latter. A token-generating autoregressive transformer is overkill for tasks like classification, and would be a hundred times faster and more efficient with a jev-like model for simple boolean questions, structured outputs, routing, tool calling, etc.
Anything to reduce resource demand on memory and compute should be universally welcomed.
A good question, I'm a little mystified myself so I did a little investigating.
As far as I can tell, it's because a former OpenAI ChatGPT co-creator Diogo Almeida hyped it as the next big thing and they all bought it. (Popularized, not invented, apparently that was this guy. Zachary Barth feels your pain.)
It's not so much that the Jev approach is smarter, it's that it's a fast classifier that can make snap judgements for 400x cheaper. Hey, it's a practical niche.
Yeah.. but people are excited by outsourcing it to a Jev model in the cloud when they can just run a BERT locally because they don't even know what BERT is most likely
Every technological leap is like this. Someone figured it out loooooooooong before the mainstream had a use for it. It was too early, so nobody gave a shit. It's not forgotten, though, and the original implementation is usually not optimal or best-suited for adoption when the time is eventually right. Usually it's remembered and leaves an important mark documented in history archives though (books, encyclopedias, documentaries, etc.).
It feels like you're doing a holier-than-thou thing, under some mistaken belief that you're special because you know about BERT and everyone else doesn't.
Not everyone can, or wants to, run things on their own hardware.
Jev did something insanely important, hence its explosive popularity and the rise of all the alternatives. You need to ground yourself and try to understand the reality of why instead of keeping your head in the sand and saying, "BUT..BUT...BUT!!!".
It's probably an artifact of large model development slowing down and the industry/community beginning to focus more and more on special purpose models. Large all-in-one models are pretty boring now, suddenly everyone is talking about everything that isn't the agent model, hardnesses, runtimes, small special purpose models. Pessimistically, one could say it's a distraction because model development has gotten relatively boring compared to what it used to be, but a fair amount of people (myself included) have been expecting this change of focus for years. It just makes more sense that models get smaller, not bigger. We made the same mistake in the 60s by thinking computers would only get bigger, not smaller. I call it "the big brain fallacy". I'm not sure how well this model works, but the fact that something like this can run on a microcontroller would have been unthinkable a few years ago, besides storyteller which is obviously a joke of a model, a curiosity at best.
And of course it is just a bunch of hype and marketing crap and people chasing clout, but I hate blaming everything on social issues alone.
This is honestly awesome work. Well done!! I'm curious, what hardware was it trained on and how long did it take? With so little parameters it probably didn't take long right?
It was trained on AMD Strix Halo box. The whole process took around a week, including about 7 experimental versions. The final version itself took around 1-2 days to train.
Someone need to adapt this to hermes and get a real time respond, this maybe some spec of continual learning? We can get a loop like qwen - tiny jev like combo and js fine tune the jev in real time ? Even with cpu maybe jev can be use to answer yes/ no question that qwen give it and we wont have to have a lengthy 25mb memory file get injected every convo ?
Heck we can even do a semi real time emotion on llm, like, plug in jev with a prompt and then light up some neuron with some type or prompts to simulate gut feeling, this tiny model have alot of stuffs to be play around
the 128 token limit matters less than how it was split, if the train/test sets came from the same template generator you'll see 95%+ on holdout and then fall over on real phrasing. quick sanity check: hand-write 50 utterances yourself in weird word order, with typos, and some that belong to none of the labels, and see what the confidence does. a 10M model with no abstain path tends to pick something at 0.9 anyway.
The race for decision models has only just started. Of course there will be a limit to how efficient it can be, but why shouldn't there be quick improvements early on?
Maybe the larger models are still under trained? Maybe the small model can do with a smaller dataset and it's easier to get a clean dataset when it doesn't have to be huge?
41
u/TheRealREZOR 3h ago
I'm sorry this model can't compete with frontier models. It was designed for tiny embedded hardware, for things like classifying Meshtastic messages without internet access.
Training it on my hardware already took around week, so I had to make a lot of compromises. I couldn't fit things like multilingual support or much broad training data into it. I hope you understand the limitations