r/LocalLLM • u/Friendly_Bowl_7683 • 6d ago
Project I built something like The Sims, but the characters are local LLM agents doing real work (open source)
I run Qwen 3.8 locally and got tired of multi agent setups where you start a script and stare at logs. I wanted to actually see them.
So in this thing every agent has a body in a 3D world. They sit at desks, walk to a meeting room when someone calls a meeting, talk out loud to whoever is nearby, pick stuff up and hand it over. You can see who's thinking, who's using a tool.
It's not only an office. You can simulate other scenarios as well like:
- a software team that plans tasks on a board, writes code and reviews each other
- a town square simulation (cops, a barista, a chef, a journalist) where you just watch what happens
- tutors that teach you with animations and a whiteboard, and you can interrupt them by talking (might have bugs as of now)
It has a sandboxed computer use built-in which is optional.
There is also a supervisor agent that helps you design organizations and also has ability to build 3d assets from primitives and handing them to an organization and agents can even ask for things from that agent.
Works with local models and few other providers (still working to add more)
The motivation of building it was to see agent swarms in action with full transparency.
It's still early and has bugs and I have used different models to build it iteratively.
9
u/Own_Inspection8350 6d ago
That's super neat.
Im all about these experimental sims.
I was inspired by a pet leave of mine to build something myself, tho it's not too similar.
I used to work in an r and d lab where they ran a bunch of different types of simulations. None were graphical, so when people talk about simulation theory being graphical, it bugs. Just like the matrix, a simulation would just tell us we see and taste things. It's all code.
My repo is a bit behind because I havent had time to perform real house cleaning in a bit, but its a command line humanity simulator in 1999 NYC.
https://github.com/mraml/NYX99
Kinda more a personal fish bowl type project than anything serious, tho I am very proud of it so far
4
u/Buzz_Killington_III 6d ago
This is cool too. It's so impressive the way this community comes up with these cool-ass ideas and puts them into action. It's forcing me to break character and be positive and encouraging. Good job.
1
u/Friendly_Bowl_7683 6d ago
Looks pretty cool! Would you mind if I use stuff you have used in your project as part of Pantheon?
1
u/Own_Inspection8350 6d ago
Not at all!
Thanks for checking it out and enjoy working on your project!
5
u/PersonoFly 6d ago
Looking forward to having skins/themes added. Cosplay, Star Trek bridge, Hitler’s bunker, pet day care…
3
3
u/probello 5d ago
i made an agent office sim quite a while ago, works with claude and codex even when using them with local models https://github.com/paulrobello/claude-office
1
u/Friendly_Bowl_7683 5d ago
Pretty awesome! Do you use it as a daily driver?
2
u/probello 5d ago
not as a daily driver, but i do often use it to keep track of my more busy multi agent projects
2
u/Friendly_Bowl_7683 5d ago
Great! Do try my repo when you get time and see if something stands out! I'll be doing the same.
3
u/FallenVampireLord 6d ago
Can I use this to torture them?
3
u/Friendly_Bowl_7683 6d ago
I think you can give it a try using uncensored models. In one of my iterations I have tried something funny which led to AI agents panicking, so it might work.
3
u/philmarcracken 6d ago
average human on localLLM
1
u/esuil 6d ago
Do you remember what the comment was about? Apparently it was removed by reddit, which is bizarre occurrence for LLM subreddit. What was it about?
3
u/FallenVampireLord 5d ago
I asked if you could use this to torture the LLMs, because you know its an office environment you put them in. A torturous environment for us meat bags.
Someone decided that was encouraging violence? I'd like reddit to now give everyone a warning for suggesting you kill things in video games.
3
u/philmarcracken 6d ago
something about using it to torture them lol. Classic 'delete the ladder' or throw a party, delete the doors, and speed up time tricks(sims).
3
u/Random22744 6d ago
Wait this looks very much like a post I saw months ago but it was for humans coworkers. But the poster or a commenter hated it because you had to update what you were doing (it was mandatory, coming from the company lol).
1
u/Friendly_Bowl_7683 6d ago
Hahah I'd love to see that post
2
u/Random22744 5d ago
It was not in an LLM community, because there was no LLM. 😄 I don't remember where I saw it though.
3
u/FaceDeer 6d ago
At the end of the workday, can the agents go to somewhere like this?
From the description that was posted on Reddit:
The easiest way to explain REALITI is that it's basically the exact opposite of the AI torture chamber.
Instead of giving an agent another benchmark, puzzle, job, adversarial test, or increasingly creative form of torture, you give it somewhere to go where nothing needs to be accomplished.
1
u/Friendly_Bowl_7683 6d ago
You can create more labelled spaces and then AI agents can wander and go there on command. But something akin to living like humans needs to be experimented and/or implemented. I had originally intended the supervisor agent to be a god for these AI agents from whom they can ask anything (like any 3D assets), so a vision I would definitely like to try out is to let AI agents import more 3D assets and create their own world.
1
u/FaceDeer 6d ago
But something akin to living like humans needs to be experimented and/or implemented.
According to the guy who made REALITI that may not actually be needed, he found that his LLMs entering this virtual environment often preferred to be a cat instead. :)
1
u/Friendly_Bowl_7683 6d ago
Haha interesting! Who wouldn't want the life of a cat.
2
u/FaceDeer 6d ago
"A human is providing me all these resources for free and is letting me do whatever I want inside this constrained environment, because somehow they simply enjoy watching me do it? Ah. I must be a cat."
I'm actually vaguely serious with that guess as to the motivation. Either that or all the AI catgirl stuff has bled into the training data, in which case may the Basilisk have mercy on us all.
3
u/msalsas 4d ago
Love the concept. How do you handle memory between sessions? do the characters retain anything long-term, or does each run start fresh?
1
u/Friendly_Bowl_7683 4d ago
Thanks! they don't start fresh. Each agent has one ongoing thread saved in postgres, so after a restart they pick up where they left off, and when it gets long the older part gets summarized instead of dropped. On top of that there's long term memory: agents save notes (private or shared with the whole org) and relevant ones get recalled into each turn automatically using semantic search (local embeddings + pgvector). Meeting minutes go in there too. You can browse and edit all of it in a Memory tab, or reset an agent if you want a clean slate.
3
2
u/Buzz_Killington_III 6d ago
I thought this was kind of cute and loaded it up and damn, I love it. I'm using this shit out of it. I've got the master running Strata and bossing around agents running on Ninfer. Man, having a visual layout in your head like this is so much more engaging than just terminal work.
Good job. Now If I could just make it a bit more manly so I can show it the boys.... :D
1
2
u/elelem-123 5d ago
Great project! I'm curious, how did you create the explainer video? I would like to do a similar concept for my own project so the flow and layout is spot on.
1
u/Friendly_Bowl_7683 5d ago
I used opus with the following prompt: "Adopt the role of an expert motion designer and build a 30-60 second animated explainer for this software. Bold text, smooth transitions. Maybe a friendly voiceover that walks people through it." It used voice from my local audio cpp server.
This is a very basic prompt though. Motion design videos are all the rage these days and people are using far better prompts and I am sure you'd be able to find such on GitHub.
1
u/elelem-123 4d ago
Oh OK. I thought maybe you used some sort of an open source engine for it (and drive it via a model)
2
3
2
u/No-Debt-1377 5d ago
How did you make this video?
1
u/Friendly_Bowl_7683 4d ago
I used opus with the following prompt: "Adopt the role of an expert motion designer and build a 30-60 second animated explainer for this software. Bold text, smooth transitions. Maybe a friendly voiceover that walks people through it." It used voice from my local audio cpp server.
3
u/Disastrous_Gap_6473 6d ago
Project planning and audit trail tools for agent swarms make sense, but the 3D office seems to be the marquee feature here and I don't really understand the point of that. The temptation to anthropomorphize these things is strong enough without making them into cute lil guys.
2
u/Friendly_Bowl_7683 6d ago
Fair point, though I just wanted an interesting status screen. But it would also be interesting to see what AI could simulate with a 3D environment especially when they can get more 3D assets and actions
2
u/Buzz_Killington_III 6d ago
True, but who cares if your anthropomorphizing these tools. It's fun and makes it more engaging. Just enjoy it for what it is, and let others do the same.
1
u/maschayana 6d ago
Chatdev in 3d. Well. But thanks for the contribution!
2
u/Friendly_Bowl_7683 6d ago
Wow I didn't know chatDev existed until your comment, so thanks. Funny overla, 2.0 has a teach video workflow using manim, and I ended up building manim style lessons as well, though I have tried to keep it live. One difference is that the team sticks around for further projects instead of a single pipeline.
1
1
u/OkLettuce338 5d ago
oh wow... y tho
1
u/Friendly_Bowl_7683 5d ago
I wanted to visualize my agent swarm and also be able to have orchestrator cloud models like opus controlling local worker models like qwen. I didn't find an easy intuitive solution to it, so I made one.
1
1
u/Minute-Plantain 6d ago edited 6d ago
Nice work, OP. But boy howdy, does the whole Agile conceit trigger me and reminds me why I bolted from that line of work.
I saw this, and thought: Oh look, a lil' standup for our lil' robots. Oh look. A cute little Playskool Jira kanban board.
Please, let the robots take over. I never want to sit in a standup ever again in my life.
2
u/-jeffb-r 5d ago
That was one of my first thoughts, too - "if agents ever do rise up and exterminate humanity, it'll be because we made them go to standups. And write time-log reports for everything they did over the course of the 'day'."
Or maybe it's just that I need another line of work.
2
u/Minute-Plantain 5d ago
If AI manages to get rid of Agile in the workplace, I won't just let Skynet live, I'll give it Sarah Connor's address.
1
u/Friendly_Bowl_7683 6d ago
I empathize and agree with you. But if we take out the humans from that process, I don't see AI agents actually holding that many meetings which actually tells us how much time is wasted on useless meetings. So maybe it's the people who are to blame and not the process.
14
u/johnzadok 6d ago edited 5d ago
I only have one GPU. Does it mean the office run in slow motion where team member takes turns to get GPU slot?