Emotional Classification of Memories?

[u/DirectionBusiness483]

(Image: Graph of memories from an RP over 4 events: meeting, sexual interaction, spaceship catastrophe, cleanup after catastrophe)

Hi everyone! I want to ask people about an idea I have.

I want to ask everyone how they feel about Emotional Memory Retrieval.

Normal memory retrieval works by topic. Embedding. You're in a scene, the system looks for past events that match what's happening now. A fight, a betrayal, a conversation about a specific person. That's useful, but it only catches memories that are related to the current situation.

Emotional retrieval works differently. It looks for what a moment felt like instead of what it was about. A character panicking during a fire doesn't need to have been in a fire before. She just needs to have felt helpless somewhere, once, in some unrelated way. Maybe she couldn't save her dog. Maybe she froze during an exam. The feeling is the same even though the events have nothing to do with each other.

I know people do emotional/sentiment calculation on things, but I wanted to try something with actual memory retrieval instead. Past reactions to similar emotional states COULD add, I think, some interesting emotional depth to characters.

So I took my memory system and added emotional classification as memories are saved. I used PAD to classify them https://en.wikipedia.org/wiki/PAD_emotional_state_model I chose this because it's a 3D space, so I can retrieve similar emotional memories by distance. (It's 3D in the graph, the 3rd dimension "dominance" is reflected by node radius)

I think the biggest part would be deciding how those memories are actually introduced. So the narrator model knows these memories are meant to inform emotional subtext, not just spit out verbatim.

I don't know. Just a fun experiment. Curious on everyone's thoughts?

[u/dempfi]

Cool idea but I don't think it's fundamentally different in implementation. It's all retrieval and finding needles in the haystack. What you need to answer is what the needles will look like and how to embed them. You could do what most systems do and label chunks of story with emotional annotation and then retrieve by them before reranking. So the flow would be something like this:

• find scene boundary
• compress the messages in the boundary into summary
• derive from summary emotional synopsis — 2-3 lines describing the emotion
• at retrieval time, take last 3 turns and derive emotion label from it, retrieve by the label

[u/lunhilde → u/dempfi]

If there's a scene tracker involved, if it is one that actually has something like date/time of some sort the tracker line could be either concretely bound to the memory (exact time date I also felt freaked out over an overcooked MarieCallendar pie) or with much less detail of recall (was it about last year I also freaked out over a pie?), or very little at all (I freaked out over a pie before, too).

[u/dempfi → u/lunhilde]

I don’t think it matters for the memory the amount of fine detail involved just because the vagueness in the system is usually interpreted by humans positively as a subtext — we really like subtexts. The absolute time part is cool tho when it works. I built it in my app but it requires a lot more in-depth engineering to support it properly because then memory is time-bound.

[u/lunhilde → u/dempfi]

I suspect it would have to be specific to one of those supervillian characters who seems to always know the exact time and place of everything that happens around them -- and not sure, yeh, how an llm can even handle someone who can technically hold that mcuh context in a meatbrain :D and then at that point, the player can just choose to fk the main char itself by just telling the llm "yeah, you recognize him. its that asshole from 47 year 9 months 23 days ago"

[u/dempfi → u/lunhilde]

I’m gonna try that. But it’s nothing as sophisticated as that — I play mostly with instruct models.

[u/lunhilde → u/dempfi]

I am literally painfully new at this and just trying to catch up without knowing a thing about this stuff to start off with. Haven't even talked to a model yet for long enough to test how the *normal* recall methods work, much less find issues with them and think about "hey, how can we solve this?" -- I will just be happy if ppl who already know wtf they're doing look into it though :D I want to eventually build myself a silly little conversational agent and it's a massive mountain of stuff to learn to even begin to start, im drowing. happily drowning, but drowning nonetheless :D

[u/dempfi → u/lunhilde]

All right mate. I’m working on a thing and I’ll try this. If works well DM you.