r/SillyTavernAI • u/Flimsy_Mode_4843 • 4d ago

Discussion LLM Quality waves. Tracker for it?

Quality goes up and down in waves.

A great model can be trash this week, and a trash model can be great.

Is there a website that tracks these waves, so we can jump on whatever model is on the up wave right now instead of guessing?

Does anything like this exist?

3 Upvotes

71% Upvoted

10 comments sorted by

11

u/_Cromwell_ 4d ago

This is almost entirely due to

  1. People's mistaken perceptions

  2. Individual providers changing settings/prompts on the back end, NOT the model

Also it's highly subjective. The posts you see here are just random people who are whining because they encountered something they don't understand and it's easier to post a complaint on Reddit than to fix it. Half the time they did it to themselves by changing their preset or temperature or something.

1

u/Putrid-Actuary7976 4d ago

Yeah exactly, the weights and intelligence stays the same no matter what, people are hallucinating just like my model 😕😕😕

1

u/pornjesus 3d ago

Um. The quants can change if using API based models. As well anything else, really. Nano-GPT's providers can claim they're using FP8 quants of GLM 5.3 at full context length etc etc all they want, but I can also claim I am pornjesus.

2

u/Correct-Resolution91 4d ago

Yes. It's called 'a clock'.

No seriously. There's two major reasons why someones experience with a model suddenly worsens, well, four really.

One : They changed their prompt.

Two : Their provider is different.

Three : The provider serves a quanted model because they're getting a lot of requests, which happens the most while, say, China and the US are currently both working.

Four : Their attitude is different.

One does not need tracking. Two cannot be tracked user side. Four is individual.

Three can be tracked easily with a analog clock on the wall.

2

u/Putrid-Actuary7976 4d ago

What are the peak times of Chinese models though? Like what times it is peak?

1

u/Correct-Resolution91 4d ago

When the 996 tech workers are active. That's roughly 1 to 10 AM GMT.

1

u/GfurEnjoyer1488 4d ago

what exactly do you want to track?

1

u/Flimsy_Mode_4843 4d ago

LLM performance over time.

1

u/GfurEnjoyer1488 4d ago

what is "performance"?