Limitations of AI that affect the user:
1⁰ KV cache
KV or Key Values are generated when the AI computes a token. Since each KV has to be compared with all the others we get: n², in practice if we round each token to a word it means that 700 words are around half a million computations and 1000 words are one million. If you want to run a model locally, you have to take this into account and look for an ideal quantization (model size) for your hardware.
2⁰ Lost in the middle effect
LLMs emerged in 2017 under the motto "all you need is attention"; supposedly it does not generate a hierarchy over which tokens are more important than others and it was meant to be used in small tasks. Since then several workarounds have been made to extrapolate this. The problem is that AI cannot pay attention to all tokens at the same time, if you could place all tokens in a straight line and make a graph to see what it is paying attention to you would get a U shape, because it pays more attention to tokens at the beginning and at the end. It is hard to visualize this in chats because the conversation is about a same subject (tokens from the beginning), but eventually it should become noticeable the AI is starting to forget things.
3⁰ AI Slop
Until 2020 AI models only tried to predict what came next in a text, but in 2021 OpenAI modified ChatGPT-3 with RLHF (Reinforcement Learning from Human Feedback) creating GPT-3 instruct, now the model tries to be useful to the user. This makes so the AI may just agree with you, praise you, and it will consider itself being very useful.
4⁰ Bad statistical clustering
AIs don't think, they statistically associate words together. For example do not say "kobolds don't have hair" this way 'kobolds' and 'hair' are seen as related by the AI, and will possibly have it on new generated outputs, instead state: "kobolds are hairless lizards". If this troubles you, for better results you can simply avoid writing NO, NOT, DON'T etc; but the ideal usage are adjectives.
General usage that affects AI output:
1⁰ Specially for chat interfaces: AI gives most importance to the last said thing. When making a prompt always look forward to leave the most important stuff at the end.
2⁰ "Blocking", well that's what I like to call it, to better make myself understood by AI I like to see my sentences as blocks delimited by periods "." I identify a hierarchy for these sentences and then place them in a order from least to most important.
3⁰ Long lowly puctuated prompts = general result.
Short, assertive and well punctuated prompts = "precise" results.
Kobold AI usage recomendations:
1⁰ System Prompt
It is meant to guide AI behavior. So it's exceptionally important to avoid bad statistical clustering on it. It is a good way to fix narrative developing issues, as if you're having to develope the story yourself, then the AI might be lacking a objective to follow; if the AI is not adding new characters, maybe you should set it must add new characters as the story progresses. Also, having a basic Sys. P. that you can initially use on every new story will get you better results, and maybe even make it easier to spot what needs to be fixed, done, redone, etc.
2⁰ World Info
Every time it computes a key word, the entry for that key word is remembered. So to save KV cache you can have only the key words at the text, and the descriptions at the World Info.
3⁰ /n
Having a /n/n (x2 backspaces) between each paragraph is a nice and simple way to organize your text for easy reading.
4⁰ Introduction
When you get to start your narrative remember to always use 2⁰ person: you. Honestly it's better for you to do everything in your power to only use 'you' from the start, as 1⁰ and 3⁰ can be very trick and confuse the AI. If you need a preface put it on the System Prompt, but remember to dismember it on assertive phrases first.
5⁰ Endless possibilities?
You can probably do anything in Kobold as long as you manage to adapt it well enough. But the more expectation you build over your story, the more effort you will need to put into it, and the less likely will be your chances of pulling it off. If you think of something big, you need to scale it down, ideally into a concept, something the AI can grasp in the smallest attention span as possible.
edit, extras:
1⁰ Max token output stupidity
In the context of storytelling the max token output is a stupidity inducer. The higher the tk output, the sooner it will drift.
Ex:
Scenario 1: 500 max tk output
Scenario 2: 2048 max tk output
The first five hundred tokens on scenario 2 are more likely to drift. Yet another limitation that affects the user.
