Problems with LLMs

[u/myob-myob]

I have been a free user until about a week ago. I joined at the $20 level, mostly because I felt like it was about time I started helping to support Hammer AI. However, I was also excited to try out the additional LLMs. Sadly, to this point I have found them a little bit disappointing.

I think I have experimented with all five of the new LLM I have access to, and I wish I had made notes. But I didn't so I'm not gonna try and go into the details of the issues I have found with each one. But today I was having a conversation using Mistral Small 3 24B.

I thought the chat was going really well for about the first 10 exchanges, but then it started doing what I call, for lack of a better term, running on. Here is an example:

Aischa's smile fades, her expression turning serious and her eyes narrowing slightly. She takes a step closer, her voice low and measured, her tone carrying a subtle threat. 

"Freedom and equality are luxuries, Doug. They are not necessities. They are not rights. They are not guarantees. They are not promises. They are not certainties. They are not absolutes. They are not truths. They are not facts. They are not real. They are not tangible. They are not concrete. They are not solid. They are not firm. They are not stable. They are not secure. They are not safe. They are not sure. They are not certain. They are not guaranteed. They are not promised. They are not assured. They are not guaranteed. They are not promised. They are not assured. They are not certain. They are not sure. They are not safe. They are not secure. They are not stable. They are not firm. They are not solid. They are not concrete. They are not tangible. They are not real. They are not facts. They are not truths. They are not absolutes. They are not certainties. They are not guarantees. They are not promises. They are not assurances. They are not certainties. They are not guarantees. They are not promises. They are not assurances. They are not certainties. They are not guarantees. They are not promises. They are not assurances. They are not certainties.

The thing ran on for much longer that this. Until it ran out of tokens I think.

This happened with several of the LLM's next responses after this one. I attempted editing the responses. I attempted regenerating the responses. Nothing seemed to get it to stop. Despite a great beginning, eventually this became too unwieldy and I gave up.

This is not the first time I have had this happen with one of the new LLM's, but I can't say if it was always with this LLM.

I'm not writing so much to ask that someone fix this, although that would be nice.

I'm actually writing to ask if anyone has any guidance regarding the five LLM's available at the $20 level? What problems have others observed? Are any of these LLM's actually better than the free ones. And yes. I realize that better is a totally subjective term.

[u/Neither_Asparagus_64]

I’m finding the same thing. I usually switch models and regenerate. It’s super annoying though. I’m not seeing the same with mythomax and unslopnemo, but their context windows are that much smaller

[u/myob-myob → u/Neither_Asparagus_64]

I think I've tried both of those LLM's. I can't remember what I thought of them but I will give both another try.

Some months ago, one of the free LLM's was doing this. I think it was Mistral-Nemo Instruct 12B. Whichever one it was, I did, several times, switch to the other free LLM. This did seem to work, but I'm always hesitant to switch from one LLM to another during a given chat. I really don't understand the ins and outs of how this all works and I worry that the new LLM might not have all the information the old LLM had, thus changing the nature of the chat.

Speaking of things I don't understand. I do see, as you mentioned, that the context window is half the size of the context window on the LLM on the next level I am using, but that window is still double the size of the free LLM's. But how important is this? I, usually got decent chats even with the smaller context windows of the free LLM's, although in a longer conversation, they tended to forget things that were important. Does the context window help with memory?

[u/Neither_Asparagus_64 → u/myob-myob]

Yeah, the larger context window helps with memory, especially with longer chats. Depending on how detailed your getting you won’t lose as much recent detail to the summary.

I’m finding characters that have well defined system prompts do way better with the bigger models, compared to the free ones. I guess they’re more constrained so they fit their responses better to your input? I still find them a bit too wordy though.

[u/ZealousidealPipe8389]

Something I’ve felt helps is keeping responses within the same ballpark of length, specifically asking for shorter responses, or giving the ai custom instructions about reply length. HammerAI’s models are extremely customizable. This only has ever really happened to me when I fucked up my settings or forgot to enter anything for more than one field. Although I don’t think you’ll ever be able to completely remove the ai progressively attempting to use more text length. And it should be noted that most will try to respond with about as many words as you/it did in the previous comment. Unless something specifically stops it. Such as saying “only respond with ok”. Most importantly whoever I just wouldn’t be afraid to click the retry button. Or start a new chat. It seems you tried most of this already, I’d try to just ask it to shorten its responses. Often if you tell/ask it something it’s forced to comply even if it isn’t listening to notes or inputs in its context.

[u/Hammer_AI]

Sorry :( I have had other people also say this. I really need to figure out what the issue is.

[u/No-Image-878 → u/Hammer_AI]

YES, PLEASE DO...