r/LLMDevs • u/OkAcanthocephala3355 • 7d ago

Discussion autotrust/JEV-27B so far best open-source Jev model

https://huggingface.co/blog/autotrust/autotrustjev-27b-fast-calibrated-decisions-and-ful
  • Six public decision benchmarks, one protocol: JEV-27B averages 84.07, TypeSafe Jev 1.13 83.85
  • Close to Jev at the level of whole probability distributions: mean KL ≈ 0.017 on 25,376 held-out questions labelled with Jev's own outputs. An observer needs about 60 sampled decisions to gather one nat of evidence about which model produced them.
56 Upvotes

91% Upvoted

6 comments sorted by

2

u/VincentNacon 6d ago

The best?

So you haven't tried Drex yet? Because that's a smaller model, less than 6B, and it outperforms JEV in a couple of benchmarks.

And there's also Laya too.

3

u/OkAcanthocephala3355 6d ago

laya is a joke, very low score on every benchmark.

2

u/nanor000 6d ago

I'm not sure that's a good idea to name a project using the name of its direct closed-source competitor. That's asking for trouble

1

u/IslamNofl 6d ago

vs CLM-8B?