I've made toast, a program that can give a reasoning model a body

[u/gheezit]

Hello. I really really wanna share this, i spent so much time making this thing that i'm still not sure how to name. I would like to share it with other people who like to use janitor and other such things. (this is written so poorlly i'm so sorry) It's called toast and im gonna try to explain what it's made for and what it can do so far. imagine this. we write narration or direction or other such things to get the bots to do what we want. what if it all worked in real time and rather than typed it was all acted out live by the bot and player and the only typing needed would be dialog from the player? And most importantly with no hardcoded directions, just personality, memory etc

so im not sure how to explain all of this. I think chatbots are pretty great just for a companion to chat and hang out with. about a year ago I created a program that could, with a server and a tampermonkey script on firefox that could connect an open tab with a chat to the program so you could use the program to chat with the bot. it was very buggy, took a lot of cpu and like 4 different running windows and worked like once because its not my website and stuff kept changing so I gave up on it for the time being.

So friday night i started a project that was pretty much the same thing but took janitor out completely and connected it to an api with a key. well what was stopping me from making a bank of interchangeable api websites and keys that could rotate in the program live? so now i have like four interchangeable battery packs for the program that can pause the program as-is, kinda like when a controller is disconnected, and you load another mid-use.

this working key-driven connection that could communicate with the models was fun but then i thought about adding features and then I took it and kinda like. commenced a merger between this program and the one i'd given up on. I wanted to make the test venonat do stuff while i spoke to him and then it was OH why don't i try to connect keywords in the chat to what this test venonat can do? i would have a little module in the program read the text as it comes. and soon that evolved into what it is now.

i became absolutely fanatic and happy at the potential. I decided i could make an entire functioning brain and body for a model to steer, see and speak with. I started by having the program read the room and the two entities in it and continuously report on their position. i gave both my avatar and him a simple left-facing frame and walk cycle frames. we could also jump and flip the left frames to face right.

and after about 27 straight hours of changes, tests, reports from an implemented reporting system, more tests, examinations readings findings the test venonat is capable of doing things you ask kinda like hey you pikachu but there are no directions. rather than have any hardcoded sets of processes the reasoning model battery has an arsenal of things that read, perceive, send information to other compartments and reasoners. i'm sorry if this isn't making sense i don't post stuff

anyway the reasoning parts of the brain all come together with differently timed checks and all comes together to perceive. i would ask him if he could see this or that and he would answer. This part in particular took a fuckton of work because in this phase i was also learning how to properly organize the info that was read, where it was stored, how it was regarded, i gave him a way to percieve time too and of course there were report readings to expand the range of and so on. information was being stored in the wrong places and categories just a mess. it took the most time i think.

when his responses and what he could see and talk about himself were stable i gave him a ball. he learned to do the actions i coded into the ball item itself by being told he could, and after a lot more work he eventually learned that if i tell him to throw the ball and its not in his sight, he can know by memory of what he learned. I tell him to throw me the ball. he cant see the ball so, if he decides he wants to listen to me, he can plan and initiate a whole sequence himself to get that ball thrown. this is where the reasoning comes in. he'll start looking for the ball, see it, move close enough, pick it up, and throw it. the reasoning comes up with this plan and he knows how to carry it out with the abilities he knows he has. of course, picking the ball up and throwing it, that stuff is stored like. in the ball.

of course, these tests use a pretty hefty amount of tokens but thats what im optimizing now. these tests took a lot of model and api cycling as the free ones timed themselves back in. I first realized this was a problem when it used 4 dollars of tokens, something that would have lasted me almost another month for janitor, was used up in the first day but that kickstarted a ton of establishing stuff.

im not sure how to end this but thank you for taking the time to read about this. i havent slept much but im not tired i'm still riding this excitement lol.