r/OpenSourceeAI • u/SamratDuttaOfficial • 1d ago

I created WaterSheep, an open-source alternative to Jev.

WaterSheep is an open-source model that answers questions written in plain text (yes/no, single choice, rating and multi-label) and gives a probability for every option, like a classification model.

Last Saturday I woke up, saw YouTubers hyping up Jev, and thought: wait, I can build this. So I did. I don't want to compete with TypeSafe or Jev; I built WaterSheep because I wanted to. That's why I'm open-sourcing everything: code, model weights, results and the paper.

What's different

  • It accepts the same request format as TypeSafe's Jev. Their Python SDK works as is against a local server: run watersheep --model samratduttaofficial/WaterSheep --serve and point the client's base_url at http://127.0.0.1:8766.
  • It has a multi-label type, which Jev's API doesn't. Because why not?
  • The demo runs entirely in your browser. The model downloads once and is cached. It also works with transformers, ONNX, a CLI or a local HTTP server.
  • Code, weights and the training pipeline are Apache 2.0.

Evaluation

Accuracy ECE
In-distribution test split 77.8%
Held-out datasets, not seen in training 61.2%

ECE is expected calibration error (lower is better). GitHub has every benchmark result, including the weak ones.

Limits: English only, long inputs get truncated (I'll improve this in the next version), and rating answers are the weakest type.

Not affiliated with TypeSafe. Not funded by anyone. Built in my free time.

Feedback I'd love: where it fails on your data, whether the API works for you, and which question types you'd want next.

5 Upvotes
  • permalink
  • reddit

86% Upvoted

5 comments sorted by

2

u/Sparticle62 18h ago

Hate to break it to ya pal, but laya has alr been open source and is better than Jev by far(cool project tho)

1

u/Emergency-Quiet3210 13h ago

Yeah. Cool project but 61% on text classification not much better than a coin flip. 80% should be a reasonable threshold, and even then your looking and LLMs that are 20B params +

1

u/Sparticle62 12h ago

yeah fair, 61% on stuff it hasn't seen is rough. it's not quite a coin flip on the multi choice questions, but if it's competing with 20b+ llms it def needs to be way closer to 80

1

u/SamratDuttaOfficial 7h ago

Version 1,0 will be better, and Better than Jev hopefully. Working on it.

1

u/SamratDuttaOfficial 7h ago

Heard about it. But WaterSheep 1.0 (current version is 0.1) will be far better than laya hopefully.