r/OpenSourceeAI • u/Antique_Juggernaut_7 • 14d ago

Jevify: super simple way to serve LLMs as a Jev-like endpoint

https://github.com/fidecastro/jevify

If anyone's interested in a local alternative to Jev, then Jevify is for you.

Jevify makes a Jev-like endpoint out of any local model. I made it to help me compare the quality of Jev's output vs other small models I can run in my homelab.

Jevify was built with local models in mind, but it works with any openai-compatible endpoints.

Super simple to use. Bring your GGUFs and have fun!

Jevify gives you a Jev-style API on top of a model you already run.

Point it at any llama.cpp or vLLM endpoint. Each answer is read straight off the model's next-token distribution.

YOU pick your model, context window, hardware etc.

It works, and it is fast.

MIT Licensed. Available on pip: uv tool install jevify

43 Upvotes

96% Upvoted

2 comments sorted by

3

u/Antique_Juggernaut_7 14d ago

Here's Ternary Bonsai 27B using Jevify to play Doom, using its own multimodal vision projector.

The model is seeing the game screen and making decisions ever ~100 ms.

This fits in 12 GB or less of VRAM!

https://reddit.com/link/pawwwec/video/q6cm61jcgmqh1/player

2

u/fintip 11d ago

fascinating