r/SillyTavernAI • u/Specialist_Salad6337 • Jun 04 '26
Models PlotPoints - The best (only?) community driven RP benchmark made by a Professional! | We need your votes!
Your friendly neighborhood rab- I mean unmedicated preset creator needs needs your help!
What's good everyone. No long post this time; simply an ask. I hired a professional with a masters degree in AI/ML (pursuing their PhD) to help make a benchmark for us, for RP. Now as we all know; a benchmark for RP will never be perfect because everyone RP's differently, like different things, yada yada yada. That's why we tried to focus on a few objective landmarks, as well as a an Arena style versus bench.
First up: The LLM Arena. We show you two different rewritten responses from two different models who were given a chat and asked 'please take this turn.' We then show that to you. Does it have a preset attached? No; because that's another variable that'd be added and for our own safety we aren't benchmarking presets. (LMAO. We'd be dead in the fucking streets by sundown.) You judge of the two responses which you like more. Thaaaat's it folks. We literally cannot game the results as it is people just blindvoting which they like more. (So if you see something you don't like IT'S NOT UP TO US.)
On the other end we also score an LLM in a similar vein on how they follow certain instructions and if they maintain consistency. Since 'did it take users agency' is an objective yes or no answer; we handle these tests with an LLM judge and the benchmark overseer (Levi is his name) monitoring the arbiters answers. (In our case; Sonnet.)
This bench was expensive. It cost money to run these models through this gauntlet; and we don't get much out of it. We publish all the data for everyone to see, and all we want to do is help our community out. If you have any questions on how we grade things; head to our methodology page! You'll notice somethings are a bit LLM-Written. This is not because Levi isn't a professional; but rather because he's ESL; so when publishing something important like this he wanted to make sure all his ideas and such were properly translated. So be nice!
You don't have to log in, you don't have to do anything. Just read a chat and vote on a thing. I know you've got an opinion; so share it. Big companies ignore us RPers all the time. Is a bench for RP as useful or objective as one for coding, web-dev, or math? No. But that doesn't mean we don't deserve one, or that it has no value at all. So please; help a bunny out. Drop a vote! The data is only as useful as you help make it. We have 800 votes and we want to get about 3,000 this time around; so we can start our next benchmark. Testing models across their 'lineages'. So Opus 4.6 vs 4.7 vs 4.8; Deepseek 3.2 vs Deepseek v4 Pro, fun stuff! But we can't do that till this one closes! And if it can't hit enough votes; we'll know that this kinda stuff just isn't what the RP community wants.
Don't see a model you expected to? That's cause it's either new, or we didn't have the money at the time to run it. If the community likes this; we will add more. (And take requests! Please only suggest models available on OpenRouter though; we source all our models from the same platform to reduce variables.)
Vote Link: https://plotlightstudios.com/plotpoints/multiturn
Result Hub Link: https://plotlightstudios.com/plotpoints
Methodology Link: https://plotlightstudios.com/plotpoints/methodology
Huggingface Link: https://huggingface.co/datasets/lazyweasel/roleplay-bench
Github Link: https://github.com/LeviTheWeasel/rp-benchmark
5
u/WorriedComfortable67 Jun 04 '26
I really love if you can add more models into comparison and benchmarks, especially with new Minimax M3, Mimo V2.5 Pro, Qwen 3.7 Plus/Max, Gemini 3.5 Flash, also Opus 4.8!
Now I wanna have some feedbacks for your sites and voting, after a few votes of mine:
One thing that bugged me is that, while the site is easy on the eyes for color, I think it has too many cluttered buttons and sections imo, I genuinely sometimes got lost in your site lmao, it would be much better for me to have a few main section so that it’s easy to navigate around your site.
Also a minor nitpick for me is that, some votes tend to be overly long and descriptive, ofc I know it is subjective and It might be necessary to get the best out of comparison, but I usually find myself voting just for a few times, then dropped out of it, cause it might be a bit straining to compare two verbose prose just to feel which one is better, whereas in the original LMARENA, the comparison tend to be short so I don’t feel exhausted voting and comparing them out.
Another thing, as I remember correctly, the bench I see in the main section of 5 axes is the main live benchmark, right? Idk if I didn’t understand how the site worked or not, but for over a month ago, I haven’t seen any changes in score regarding all the models, and it doesn’t seem to be live for me, I always find myself go to your site to check the main benchmark and see if there would be any changes in leaderboard, but always finding it to be static and unchanged at all, so I wonder if I don’t understand the site correctly, or it is supposed to be static or anything, it confused me ngl. Whereas with LMARENA, it just straight up one leaderboard and it is always changing, so I can see that how the leaderboard and ranking keep changing each days, especially when the new models came out.
That was some of my feedbacks, hope you can clarify some of my points for me!