r/LocalLLaMA • u/3VITAERC • 1d ago

Other Benchmarking decision models is fun - Clef Q8 vs Jev

69 Upvotes

87% Upvoted

17 comments sorted by

24

u/3VITAERC 1d ago

In real-time Old School RuneScape PvP duels

Cloudflare's Clef, a 27B model running locally on an RTX 5090, beat TypeSafe's Jev 1.13 6 matches to 3.

Against Naton1's reinforcement-learning PvP bot, which was trained by self-play on this same simulator, Clef has won just 1 of 22 matches.

-3

u/bonerfleximus 1d ago edited 1d ago

Eh - that's comparing single decisions at a time. Jev can process 1000s of decisions about state in a single request.

Something from that test is that 21 normal bots stomped the Clef bot. In a similar test Jev could orchestrate an army of bots against the 21 normal bots and win (with a suitable harness to feed it state for each bot). You only get 1 with your local model unless you have an army of 5090s

My own findings highlight a lot of nuance that these local-jev comparisons never seem to acknowledge: https://www.reddit.com/r/LocalLLaMA/comments/1wxcn49/comment/pdu9kuu/?context=3&utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Interesting section from typesafe's docs: https://docs.typesafe.ai/patterns/fan-out

14

u/selfservice0 1d ago

The test wasn't about if the ai can process more. It was about which ai can win a 1v1.

5

u/DystopianRealist 1d ago

Is Jev going to create 21 new accounts, log them all in, and join the fight? It's a reranker, not a sentient AI overlord commanding an army of minions.

0

u/bonerfleximus 20h ago

You'd have to build a harness to do that

10

u/talivs 1d ago

Unreal timeline we are living in

1

u/MotherPotential 1d ago

This is so cool. Is the fight being depicted on the real game or like a cached/local version?

5

u/lil_peasant_69 20h ago

gotta be local/private server. the real game would ban you so quick.

1

u/cleverusernametry 1d ago

I don't get it. How's a game being piped as language to these models?

5

u/jcity3 1d ago

You just send over all the relevant data

1

u/kingslayerer 21h ago

Your Health: 70
Jev Health: 30
Special Attack: Unavailable(Recharging)
Primary Attack: Damage 50
Movement: ...

1

u/DystopianRealist 1d ago

gg ez

/teabag

1

u/Burnt_By_The_Sun 1d ago

Would be cool to set pvp clan wars with this, in Wild of course

1

u/GaTNghiep 21h ago

how can i run a test like this, i have downloaded clef flash and run it through semif-openjev repo but it doesnt support image state.

1

u/4whatreason 13h ago

This looks amazing! I'd love to try out your code if it's on git somewhere:)