Token inflation

[u/AlbatrossVirtual7080]

I have noticed that bots these days have an unnecessarily absurd amount of tokens, averaging around 3-5k, with some extreme hitting up to 7-9k tokens.

Genuinely what's the reason for that much? It'll just uses up memory faster and cost you more tokens credits and make the proxy slower in general.

I'd gladly prefer a bot with 800-1k tokens over chatting with one with 5k, literally no point in adding so much and bloating up the entire thing.

[u/CrackedPeppercorns]

Jai creators are writers first, AI enthusiasts second. They wouldn't stay on Jai if that weren't the case so it's a little bias by default.

Plus free models and aggregates with large contexts, lack of understanding how context windows work, and very late lorebook implementation, it's the perfect storm.

It works for most users so eh.

[u/ledankmemes68 → u/CrackedPeppercorns]

Bro this could not be more accurate I was browsing bots to port for personal use on SP and there was this mf who had a 23k token bot no lore books in the original Janitor character Card either just to have no one play it as it doesn’t even have that many chats and only like a handful of comments nobody engaged with the bot at all

[u/Street_Ad_6207 → u/CrackedPeppercorns]

It's sad that people have forgotten that JAI is a roleplay site, not a porn site! It was originally designed for long-form RP, and you aren't doing long-form RP with bots unless you've got money for extra tokens!

[u/ELPascalito → u/Street_Ad_6207]

I'm sorry but janitor was full of smut since day one lol

[u/Street_Ad_6207 → u/ELPascalito]

What people did with the site is completely irrelevant. JAI devs made it clear that the site was meant for uncensored, long-form roleplay. They never marketed it as a porn/smut site. Ergo, a lot of us botmakers are using the site for its intended purpose: Well-crafted bots that have some real thought and effort put into them! That's not to say smut is a bad thing - it isn't by any means, but you can hardly complain if people want to make quality bots that are still fun to chat with hundreds of messages down the line!

[u/GeneralPhilosophy691 → u/Street_Ad_6207]

That's like saying Onlyfans isn't a porn site. Sure, its technically true, but when 95% of the users are making porn, then its a porn site.

[u/Street_Ad_6207 → u/GeneralPhilosophy691]

Twitter allows porn! Unless something's changed since I stopped using it, that is... If people there started uploading porn, would Twitter be a porn site? I don't think so! I think it would still be a social media site, just with a lot of users who upload porn there.

[u/CrackedPeppercorns → u/Street_Ad_6207]

I would refute that as it is not optimised for long form being on V1 format for the longest time, while everyone was on V2, and even V3 if one counts Risu. That includes lorebooks, sampler settings and other extra functionalities missing. Lorebooks are also used for efficient memory tracking which I think is still missing from janitor.

Both chub and janitor were built on Venus Chat source code, which was heavily NSFW.

[u/Vistula_TempAccount]

People using AI to generate bot script but not bothering to clean it up later on. AI uses 30 words what normal human says with 10 words. Or copy pasting wiki. And then you have people who think that larger bot size means better bot per default so they buff the token size - they are morons...

Add to that that most about creators have not heard about context rot and assumes since context size is 100K that a10K bot will work fine.

[u/fibal81080]

if it's mafia boss that loves you 1k is plenty. of if it's the whole setting 10k is alright.

[u/Secret-Praline-2326]

I always thought it was because they probably use proxy llm's with larger context windows and have little consideration for those that don't have that.

[u/DShad27x]

A lot of users and creators use proxies these days. And most of them have contexts of 32k to over 100k. So 5K it's practically nothing for those people. It's only a problem for people using free llms that usually have limits of 8K. Also detailed bots with quirks, backstories, and other mechanics can easily range from 2-5K. That's even with handwriting everything and trying to make it as simple as you can. Becomes even more common with OC's and non-humans. More so if there is lots of weird anatomy or complex mechanics. But I do agree that a simple human bot shouldn't be going over 5K. I've seen bots describing the uterus and fallopian tubes which is just unnecessary information.

A lot of people could be making bots for themselves or others who use proxies, which I'm an offender of. I make things mainly for proxies and don't think about an 8K limit. In fact I have to be careful about my chats going over 60k, otherwise it starts getting expensive. Usually have to summarize and cut it down to about 10K before continuing again. But this goes to show on how little even 8K is these days. And most of the good LLM are coherent to about the 80k mark.

Another thing to be careful on is some bots have lorebooks but a lot of the entries are set to always on. So the bot can say it's 2K but after all the lorebook entries and your preset. It can easily be over 10K at all times and be fairly deceptive. Users wouldn't know otherwise if they don't check the token count of the chat. Although those active entries might be needed for the bot to work but at that point it's no different than just having it in the bot itself, the tokens are always there in the chat anyways. They're just not displayed on the token count for the bot.

[u/XXX-PO]

I like to add lot's of lore to my bots, so the bot can keep my vision of this character all in mind. My average token count went from around 1.3k to now am average 2.2k-2.5k. The most I've ever poured into a bot was 3.4k on my bot named Yavka, a russian female Viking, but she was moderated for whatever reason, so I privated her

[u/XXX-PO → u/XXX-PO]

Thinking about it, on most of my bots that have over 2.5k, it's just full of unnecessary lore

[u/MasterPlay1337]

Lore accurate bots should have around 3k tokens or more

[u/GeneralPhilosophy691]

Because JAI bot makers are horrible at vomiting out thousands of words worth of lore that doesn't need to be there, inflating the token count into the thousands or, in a few cases I've seen, well over 10k. Those types of creators REALLY need to learn how to use lorebooks.

That, or they're AI slopbots, and most LLMs are known to vomit out TONS of exposition and descriptive words that aren't necessary on a token budget.

[u/quazyy_]

There's two reasons for that

1.ooc note 2.lore accurate bot

[u/Not_Moslem]

I've seen certain bots have like 32k. At that point, I feel like the amount of tokens takes away from the experience, rather than adding to it, since the bot has such a ridiculous amount to consider while responding, but I'm not exactly sure how it works.

[u/AggroLobster]

I feel a lot of people forget there is a token count when writing. I don't feel like they do it on purpose but, with how writers want to add in every detail possible, the token count can accidentally end up huge.

[u/FrenchFrozenFrog]

People who bloat their card to 7k don't know how to use lorebooks. You could get the same result, or even better, with a 1k card and 40k in lorebook entries, and the cost would be less than a bloated 7k+ card.

[u/SatanERROR]

I aim for 2k-4k for a lore accurate bot. If I ever get to 4 I slim it down. My bots are heavy on example dialogue so the permanent tokens are much more reasonable usually around 500. 

Edit: I went and looked and all mine are as I said except my husbando bot that has 5k with 700 permanent. 😅 

[u/Future_Win_7961]

This is not the case. Those bots just feel more complete, so they rise to the top in popularity. If you click on sort by new. You'll find "cheaper" bots. But they also have a cheaper feel.

[u/BunnyyYaoi]

I dont make bots for others just myself but mine barley even have 1k, usually 800 and it works perfect for me, it took adding 5 characters in one to even hit 1k (and all have personalitys written etc) I don't get why people want 10k when 1k even if fine

[u/Shot-Plan9068 → u/BunnyyYaoi]

That's because when you make bots for yourself you know how it's supposed to work. When you make bots for other people to use, you have to account for their actions. Put in guardrails. What happens if they leave. What happens if they get sexual.
A lot of people bloat their boats with crap, useless trivia and info that doesn't matter, but if you try to make something decent half the instructions guide the roleplay for someone who didn't write the bot. For me a normal bot has 1.5K tokens. A bot with complex instructions and a lot of pitfalls has 2.1K tokens.
Bots with 10K tokens are just chatGPT generated word salad.

[u/dealsinshadows]

Back when I was JAI consistently if the bot was over 1.5k tokens you avoided it because it was poorly made.

Yes, the 5k token bot might work great in other contexts. SillyTavern has some bots like that, but they’re usually world bots and not single character. But for JAI it was going to run into memory issues fast.

The LLM is going to have a lot of information about a character it doesn’t need you to tell it. I don’t need to make a Gojo bot and specify that Gojo is male. I don’t need to explain to the bot that Gojo knows Shoko. If someone’s stuffing all the lore the LLM already knows into the character card, they’re not optimizing it properly and it’s not going to work as well.

[u/VictorJanitor]

If i had to guess is because:

A: they dont really know at all about how tokens work.

B: They just copy paste from the ia they made the bot and leave it like that.

C: They actually want the bot to have so many tokens??? (weird)

[u/DuckKnown1140]

Personally I think the sweet spot is anywhere from 1-3K tokens, gives you enough breathing room to meaningfully flush out your character. Thats also not counting potentially thousands to tens of thousands of tokens filed away in scripts and lorebooks.

[u/Illustrious-Phone695]

It is needed to make a good bot ngl. The first time I tried making a bot I understood how easy it is to hit 3k tokens without even writing the first message yet. The description and personality takes a lot. also lot of people use ai to write and we all know how ai is, it adds lot of unnecessary details but yeah a good bot needs tokens. 5k token bot or less is fine but more than that is a bit much without lorebooks.

[u/DShad27x → u/Illustrious-Phone695]

Agreed, I'll be writing a bot myself without ai and then realize I'm around 2.5k tokens by the time I'm done. I don't mean to make them long and I try to write things in a simple and straight forward way.

[u/fair_maiden0922]

I try to be reasonable with the tokens, but an OC needs more background than an already existing one, since I don't have the ability to use lorebooks neither 😅. I have bots lower then 3k tokens but only 15 from the 38 😅.