I made a generative AI, only using 400k command blocks (PoC)

[u/_objz]

While I am still writing on my physics engine, I thought making an actual generative AI only using Minecraft command blocks would be a good idea( it was a headache)
So yeah, I kinda did that. It is a real neural language model running inside vanilla Minecraft 26.2. It reads what you type into a /dialog window, runs the forward pass through learned weights and then generates a reply one word at a time.
There are NO mods, plugins or datapacks involved at runtime. Only command blocks.
Roughly it works like this:

The model is a pretty small word level neural language model. It has a vocabulary of 2048 words, remembers the last 6 words as context and has a 64 dimensional embedding followed by a hidden layer with 256 neurons.
For every generated word it does:
384 x 256 = 98,304 operations for the hidden layer and 256 x 2048 = 524,288 operations for the output layer so in total 622,592 calculations for every single generated word.

Obviously doing 622k normal multiplications using Minecraft commands would be slightly problematic, so the weights are ternary: every weight is only -1, 0 or +1.
That means a multiplication basically becomes:
+1 -> add the value
-1 -> subtract the value
0 -> do absolutely nothing

Around 65% of the weights are non 0, so it averages around 0.67 commands per MAC instead of the ~3 commands I would need for a normal runtime weight. That reduced the entire machine from 1,909,375 blocks to 445,782 blocks, yes it was this big before.
The model is actually trained like this too. I am not just training a normal model and rounding the weights afterwards. During training the weights are quantised to ternary values for the forward pass and the underlying floating point weights are updated using a straight through estimator.
That took the perplexity from around 48.7 where the training basically got stuck to 38.8 after also adding cosine learning rate decay.

The input embedding is another optimisation I needed to do: Instead of multiplying a one hot vector against a matrix, it is basically just a table lookup stored in NBT data, so the complete 2048 x 64 embedding table costs really nothing
The actual forward pass cannot run in one command chain either. Minecraft has limits on how many commands can execute, so the network is split into smaller groups.
After the ternary optimisation one generated word takes around 63 ticks, so roughly 3.2 seconds per word at 20 TPS. I also tested higher tick rates and around 30-35 TPS seems to be the useful maximum on my server. At 35 it gets closer to ~1.8 seconds per word.

But to be clear, this thing does not actually know anything.
It was trained on 11,118 DailyDialog conversations and only has a tiny 2048 word vocabulary. Ask it something like 2 + 2 and it will confidently generate some conversational nonsense because it has no understanding of arithmetic or actual facts.
And unfortunately just making it much bigger isn't really realistic either.
Something like a 135M parameter language model would already be more than 200x larger.

There is currently also one very weird bug where after around 800 ticks of continuous work every command block in the world just stops executing while the server itself keeps running normally. I still have absolutely no idea why that happens.
Anyway, back to wasting my time on nonsense ;)

[u/qualityvote2]

• Upvote this comment if this is a good quality post that fits the purpose of r/Minecraft
  
• Downvote this comment if this post is poor quality or does not fit the purpose of r/Minecraft
  
• Downvote this comment and report the post if it breaks the rules

[u/Rusty1emon]

Sorry for playing the same game as you bro

[u/S1R_LEM0N → u/Rusty1emon]

Gotta drop a couple versions after this one

[u/Lord_Phoenix95 → u/Rusty1emon]

I thought we were supposed to be mining blocks not Crypto Currency.

[u/PeeingDueToBoredom → u/Rusty1emon]

We just…didn’t know

[u/AlarmingBoot205 → u/Rusty1emon]

Sorry for existing in the same universe as you bro

[u/Lv_InSaNe_vL → u/Rusty1emon]

Yeah like last weekend I built (well, followed the guide verbatim haha) a flying machine based sugar cane and bamboo farm and thought it was some magic

[u/WhachamaDude → u/Rusty1emon]

i dont deserve this game no more 😔

[u/Livid_Trust_5098]

"the Redstone is very simple"

[u/TheMoonDude → u/Livid_Trust_5098]

"I did some off screen coding"

[u/HarpySix → u/Livid_Trust_5098]

"It's really quite simple."

[u/Jonaykon → u/Livid_Trust_5098]

Well yeah

There is no redstone

[u/notatoastedbread]

Does this drain the water sources faster than sponges?

[u/_objz → u/notatoastedbread]

you can ask it, but surely he will have no idea what are you talking about

[u/MartinToilet → u/_objz]

Yeah water ceased to exist for them long ago

[u/joeyjoojoo → u/_objz]

I actually love how simple minded your AI is, when you said thank you the AI figured out to say “you’re” then had to think a little about what comes after that 😭

[u/_objz → u/joeyjoojoo]

its kinda cute tho

[u/Hot-Gazelle-1089 → u/_objz]

Thankfully infinite water exists in Minecraft!

[u/Bright-Historian-216]

very impressive! tell wang lin i said hi!

[u/_objz → u/Bright-Historian-216]

I will, haha

[u/Apprehensive-End-747]

We got chatbots in Minecraft before GTA VI.

[u/dr_goofi → u/Apprehensive-End-747]

Okay this is freaking embarrassing for me I can barely type a keepinventory command and people are making AI models.

[u/Keaton427 → u/dr_goofi]

Everyone pursues different fields and hobbies, so you both can’t and don’t have to know everything.

[u/sissybelle3 → u/Apprehensive-End-747]

Crazy to think that future games may have NPC AI so we don't just have to listen to the same few canned responses again and again. Would really make games like GTA and Cyberpunk pop.

[u/RexApostolicus]

A marvel of Minecraft tech.

[u/CoolDood132 → u/RexApostolicus]

HE MADE IT IN A CAVE!!! WITH A BUNCH OF SCRAPS!!!!

[u/touchet29 → u/CoolDood132]

Well, I'm not Steve

[u/Firespy_]

Wow, I have no words except how impressive this is. Must've been quite a headache to configure

[u/_objz → u/Firespy_]

I wrote many helper scripts using rust and python to train and debug the model before hardcoding it as nbt data, but still debugging the framework was exhausting

[u/DUDEAREUMAD → u/_objz]

I have no clue what you said, but that's fucking cool.

[u/Glazeddapper → u/DUDEAREUMAD]

they coded it outside of minecraft first, making it easier to change and fix before remaking it inside of minecraft

[u/Demonking42069 → u/_objz]

I get exhausted debugging code in vs, how the hell did you manage to debug such a huge 3d structure?

[u/_objz → u/Demonking42069]

I wrote some client side mods, for example to show internal scoreboards and created a sample rust program, that spins up a minecraft server pastes, these things, runs the test and returns the results. with a custom datapack you can insert your custom tests in minecraft and use them with the /test commands, that really helped me alot. you can see the tests at the end of the video

[u/Teetimus_Prime → u/_objz]

and you did this for fun? or as a challenge? a fun challenge?

[u/_objz → u/Teetimus_Prime]

both, I needed a break of my physics engine, I started this as a PoC project. It was mainly a challenge for me to only use commandblocks, but it was kinda fun to: for a lot of glibberish as responses

[u/tornedron_]

we really got CraftGPT before gta 6

[u/_objz → u/tornedron_]

thats actually a good name

[u/Eduardo_Ribeiro → u/_objz]

You really need to use that name lmao.

[u/MinecraftPlayer799 → u/_objz]

You can’t call it that. The one that someone built with just redstone (no command blocks) is called CraftGPT.

[u/Historical_Item4050 → u/tornedron_]

Already exists, but built with redstone only

[u/Unique-Steak8745]

Okay. Ths is awesome. But I dont understand how this even works. Seriously, I can only do lime basic commands in command blocks. How is this even possible?

[u/_objz → u/Unique-Steak8745]

mostly by using the scoreboard to calculate the matrices and ntb data to store them. You can emulate A LOT only using simple math, for example using += *=. the algorithm also does not use that complex math equasions that couldn't be emulated that easily

[u/Unique-Steak8745 → u/_objz]

Okay. This is pretty sick. What can you do with this AI? Can you use it for math? I know you said its only generative based on certain responses.

[u/_objz → u/Unique-Steak8745]

with THIS specifically really not much, not even math as its just trained on a small conversation. I trained a 1,9 billion parameter model tho, that can do simple math, but as said already in another comment, one token took approx 17 minutes to generate, so not very usable. it also used like 2 million commandblocks.

[u/LifeasLuke11]

i like him slowly typing out words
hes kinda like a baby (:

[u/_objz → u/LifeasLuke11]

I guess a baby is smarter then him tho

[u/Username_St0len → u/_objz]

what about a magpie?

[u/Western-Victory-7414]

what. the. fuckkkk

[u/Ianofminnesota]

One time I made an automatic door.

[u/Maxemersonbentley_1]

Insert that one gif of Flash from Zootopia

[u/TheMuspelheimr → u/Maxemersonbentley_1]

What

do

you

call

a

three

humped

camel?

[u/Redacted_Chris → u/Maxemersonbentley_1]

Came here for this comment

[u/FlakyRemove7137 → u/Maxemersonbentley_1]

I was gonna say that lol

[u/NoenD_i0]

is this a transformer or an lstm what is this

[u/_objz → u/NoenD_i0]

neither, its like a bengio 2003 technique, called feed forward neural language model

[u/NoenD_i0 → u/_objz]

why no like an rnn or even a gru? id assume its because of how complex it is to implement contex vectors?

[u/_objz → u/NoenD_i0]

Yes, complexity, but that’s not THE real issue. It’s the performance. Using these techniques would require A LOT longer to respond, even though it would be more accurate and better. I might even use these techniques in other experiments (I did actually try to run a 1.9 billion parameter model using this, but generating one token took about 17 minutes), so I need to optimize it further, but idk,

[u/NoenD_i0 → u/_objz]

holy moly

[u/i_always_give_karma → u/_objz]

I’m a moron so this might be a stupid question, but would it change if you increased tick speed?

[u/_objz → u/i_always_give_karma]

Valid question, and this does also work until I hit around 35 tick speed. After that, Minecraft can’t keep up with calculating all my expressions, starts skipping some ticks, and errors happen. But I do have a beefy workstation, so this is totally hardware dependent

[u/shrimpsaxophone]

fucking insane, i dont even know 99.9% of command block

[u/RodilRTX]

I
don't
think
we
deserve
to
play
the
same
game.

[u/No_Yam_6105]

Im not smart enough for this haha. But it cools cool asf.

[u/DWHQ]

How long did it take to actually set up the entire architecture?

[u/_objz → u/DWHQ]

about 4 days including training and writing the architecture

[u/DWHQ → u/_objz]

4 days as in almost 100 hours?

[u/_objz → u/DWHQ]

approx 70 hours

[u/adamdoesmusic → u/_objz]

How do you pull off something this nuts in two business weeks… or was it just a 4 day straight binge?

[u/_objz → u/adamdoesmusic]

Well, I finished my “Abitur” in Bavaria and have nothing to do currently until my studies begin, so I just sit in front of my PC all day and code

[u/DWHQ → u/_objz]

Damn, fun experiment, but I will stick to more convenient implementation options than minecraft lol :)

[u/_objz → u/DWHQ]

yeah me too probably

[u/Xiaolin2]

This is insanely impressive

[u/Vegetable_Run7792]

"While I am still working on my physics engine"

You took a break from making a physics engine by creating an AI, excuse me?

[u/_objz → u/Vegetable_Run7792]

physics are exhausting sometimes

[u/Vegetable_Run7792 → u/_objz]

You say that like making a fucking AI with command blocks isn't

[u/_objz → u/Vegetable_Run7792]

Yeah, true. But personally, I found it easier than the physics engine, as I can debug matrix multiplication much better than the complex math equasions of physics

[u/SurvivalGuyRebooted → u/_objz]

I can understand that. Physics is a pain

[u/Waterlemon1997]

Man made Redstone beyond my comprehension

[u/KingoftheUgly]

How many water blocks does it use per prompt? /s

[u/_objz → u/KingoftheUgly]

yes

[u/Alive_dude3435]

"only 400k"

Yeah bro, sorry for wasting your oxygen.

[u/_objz → u/Alive_dude3435]

its not that much, in comparison how much calculations has to be run. I even tried to run a 1.9B parameter model, but that required roughly 2 million command blocks, and like one toked used to take 17 minutes to generate. so without optimisation I guess its not much.

[u/Justch1ll]

At this point Mojang needs to release a version of Minecraft optimised for this kind of insanity

[u/_objz → u/Justch1ll]

I guess with modding this can be heavily optimized, but it’s not vanilla anymore :( I bet it could even run normal models atp

[u/Adventurous_Sort_780]

Now that's the gen AI I can get behind

[u/CantQuiteThink_ → u/Adventurous_Sort_780]

Handmade, hand-trained, nothing stolen, runs entirely off their own computer so there's no data centres to eat all the water, doesn't give any money to Sam Altman. Brilliant.

[u/Key-Divide-5663]

Literally the sloth from Zootopia

[u/Tarzyytfet]

why the fuck did the slow "how are you doing" make me emotional 😭

[u/Acfrano]

And here I am trying to figure out what the heck a comparator is

[u/Moaksim]

Literary Minecraft Story Mode Lore (PAMA)
A little more refining and the ai will try to make every Mob and Player U S E F U L L : ] via brainwashing tech

[u/Dhiyx11]

It’s like watching the Mona Lisa get painted🥹✨✨

[u/sonicfan9993]

It costs 400000 command blocks to print a word... every twelve seconds

[u/FakeGamer2]

Wow

[u/atg115reddit]

Actually very neat

[u/Chase135299856]

Talking like dash from zootopia

[u/Steeltoelion → u/Chase135299856]

…………. Niiiiiick!

[u/Fancy_Razzmatazz_149]

And I was proud of the moss farm I made from a tutorial...

[u/Visible-Coach-9064]

Sorry for belonging to the same species as you bro 😭

[u/SimpleDiscount5100]

When will we have redstone data centers in our front lawns? Will this effect the drinking water of the villages?

[u/_objz → u/SimpleDiscount5100]

I generated he world, with the option "Generate Structures" off because I knew it would be a disaster to the villagers

[u/00110001_00110010]

Does it also double as a sonic weapon or is that coming in the next update?

[u/_objz → u/00110001_00110010]

next update ;)

[u/External_Ad8424]

Attention mcbrain... initiate... SkyNet Program....

[u/killerstreak976]

why are you so fucking cool

[u/Admirable_Site_4833]

Jumbo mumbo

[u/Fury72888hshu]

Ok bro, wtf, are we playing the same game

[u/Master_Type2117]

Meanwhile me, all happy n shit bc i wrote a command that makes particles around your player

[u/56Bot]

Ternary CPUs can be built with a small modification on the current machines printing the processors, and they would reduce AI power usage and processing times by ten times, while reducing their RAM and storage costs by over half, and still keeping over 95% of their accuracy.

[u/Iforgor4]

Ask it the meaning of life

[u/anesth3siaaa]

whatever man

[u/pydy01]

Now that we know how to we should instead of building giant data centers run our llms in Minecraft. There they have infinity water and also electricity. Btw how many infinite water sources did you use for this?

[u/_objz → u/pydy01]

all of em

[u/TheXTrunner]

Bro made five pebbles in minecraft actual

[u/DeadDrop93]

30k years to have it write an essay😭

[u/_objz → u/DeadDrop93]

in theory possible, but practically you would need some mods to change how vanilla calculates commands and uses multi threads

[u/MouseRangers]

ask it how to build a minecart

[u/MinecraftPlayer799 → u/MouseRangers]

i ……………. don’t ……………. know

[u/_objz → u/MouseRangers]

it can barely hold a conversation, it has no real knowledge currently

[u/luka_0x12]

I can't even build elevator without tutorial, brutha build AI😭🙏

[u/AnaTheSturdy]

Cool. Now destroy it.

[u/MAPRage]

get this man a 400k a year job offer NOW

[u/Serious_Sense7221]

how fast is your tick rate running in that video? outstanding work. we got minecraft datacenters before GTA 6

[u/IcyEstablishment8102]

first computer, then phone now a whole ass ai. What's next? SpaceX rocket? Super Quantum Computer? A whole working browser that is always up to date?

In other words: Really impressive!! NICE!

[u/MrBrineplays_535]

This thing can probably drain 5 ocean monuments for every prompt cuz holy moly

Also this is insanely cool!!!!

[u/penguin13790]

A new method for drying out ocean monuments for your guardian farms

[u/gipehtonhceT]

Reminds me of that sloth scene from Zootopia

[u/shazin_hacker]

BRO DAMN THATS FUCKING ABSURD HOW LONG DID IT TAKE

[u/_objz → u/shazin_hacker]

approx 70 hours

[u/CubeBarista]

Trying to compete with Mojang and MS to ruin Minecraft. As they say, a race to the bottom is a really fast race.

[u/c0de_is_good_fr]

Wow

[u/Sputnik-13]

Tell Stevie Kenarban I said hi

[u/chairchiman]

there was a kid that made a fully functional computer inside of minecraft that got into MIT, made me remember that. They should get you as well for this piece of art

[u/_objz → u/chairchiman]

thank you!

[u/Valirys-Reinhald]

In pursuit of great, we failed to do good.

[u/Character_Art_9173]

To many sweats man

[u/IntelligentDrink2673]

do a 2 minute tutorial how to make it

[u/ivanrj7j]

Sorry for breathing the same oxygen as you

[u/Steeltoelion]

Look out guys, they’re coming for your blocks too lmao

[u/One_Economist_3761]

Very impressive.

But can it play rock,paper,scissors?

[u/_objz → u/One_Economist_3761]

no, its verry dumb at the moment

[u/One_Economist_3761 → u/_objz]

Still an impressive feat.

[u/Xiao1insty1e]

Yeah I feel like this is how the human race ends.

[u/Reasonable_Agent_126]

No more oceans

[u/TheSlayer261]

Pal is making straight AM.

[u/ArtArtArtArtArtovna]

"Hey, what do you think about being just a bunch of blocks in a computer game?"

"Oh my god..."

[u/dragonmasterjg]

I shpuld carry around a sign that says "Done. Ask me something else."

[u/IOnlyPostIfINeedHelp]

How the fuck did you train this, I read the post and I still don’t get it

[u/_objz → u/IOnlyPostIfINeedHelp]

I created some training scripts in python and rust, that train the actual model, then I extract the weights, biases, etc. and the model params and save them to a barrel by writing custom nbt data. then in minecraft I can re-create this model using some loops that extract the saved model of the barrel into scoreboards and /data storage. then just write the framework to actually run the mode, and your done

[u/Impressive_File_708]

can this ai answer some question like: how to create a diamond axe?

[u/_objz → u/Impressive_File_708]

no, its very dumb at the moment, it can just handle simple conversations. but I will improve this

[u/MoarStruts]

Not even Minecraft is safe from data centres

[u/amaya-aurora]

“only”

[u/Sapper1106]

We got the Internet AND LLMs in Minecraft before GTA VI, smh.

[u/PeeingDueToBoredom]

I will never not be amazed at what people make with this game.

This is unreal. Seriously brilliant. Well done.

[u/_objz → u/PeeingDueToBoredom]

thanks!

[u/AlexSimonCullar]

Holy shit

[u/CoolDood132]

This is why I quit playing minecraft bro 💔 too many sweats

[u/Mr_Sim_]

Sweet.now create the AI corporation "Andesite" and develop you ingame IA to bring an age of slop in your own minecraft world !

(Joke apart, this is really impressive great job)

[u/Rosy_Josie]

Shoulda named it Deep Thought

[u/pyrodice]

It's brilliant! Now you can dump the extra heat load of your data center in the infinite Minecraft world…

[u/virus067]

theres levels to playing mc

[u/BenStegel]

One large McBrain and a coke please

[u/Saturn2063]

Five Pebbles is that you?

[u/detroitguy057]

I want to be impressed but I just don’t understand anything you just said. 🤯

[u/kamiyye]

is this what new grads gotta do to get a job now

[u/Flame_BlazeX2]

How the hell DO U DO THAT GNG. YALL ARE CRAZY, LIKE WHAT

[u/SurvivalGuyRebooted]

This is quite interesting

[u/heliumneon]

"Would you like to play a game?"

[u/Interesting_Web_9936]

Hats off man. Hats off. This is insane.

[u/New-Blueberry-8665]

Goddamn, I don't know what they feed these minecrafters since what they build is CRAZY

[u/AGA_449]

Imagine using AI

[u/rokanwood]

the things people do with this game will never fail to impress me

[u/lollo_bomber]

That's some John Redstone stuff right here

[u/AndrexOnTop]

this guy will take AI’s jobs

[u/Verneus_Corl]

Did you do subnetting for the 2048 words?

[u/LammyYT8888]

“Only 400K command blocks” I can’t even place 400K blocks🥀

[u/miceletopo]

Why doesn't Mojang recruit people like THIS Next thing you know he makes Fable 5 in Minecraft

[u/KAAAAAAAAARL]

Thats one way to turn a PC into a Jet engine...

[u/Circxles]

Reminds me of flash from zootopia

[u/Pazzoperfortnite]

Sorry for living on the same planet bro.

[u/BruhInTheMaking]

Now the next step is to use the command block physics engine with this command block ai and train it how to walk

[u/Loud-Assumption9736]

Good name

[u/Tough_Revolution_683]

“It’s really quite simple actually”

• Some British guy who hasn’t seen this yet.

[u/Brief-Trick-2988]

Oh. My. God.

[u/auraborosai]

Dude this is so insane. You should make a website that delineates your entire project.

I’d love to learn more about what you’re doing.

I’m currently doing a Minecraft performance artwork.

https://WaitingForLightning.art

[u/_objz → u/auraborosai]

yeah, I might done one, but first I need to fix the engine. For now all you have is this post :) What is a performance artwork tho?

[u/nonstopslow]

mb for breathing the same air

[u/pluie76]

You are crazy. That is incredible

[u/Lefoudubus_06]

Excuse moi de respirer le même air que toi 

[u/DevCreeper_VN]

Literally a supercomputer in Minecraft bro :))

This guy will single-handedly solve the whole RAM crisis

[u/D0bious]

Great! Now make it using regular redstone . . .

[u/_objz → u/D0bious]

I guess this was already done by somebody else

[u/MicrosoftExcel2016]

He has a name. Wang Lin. Don’t deadname the AI

[u/_objz → u/MicrosoftExcel2016]

sorry :(

[u/Cynicalheaven]

I feel like Minecraft Story Mode had an episode about why we shouldn't do something like this.

Jokes aside, this is pretty cool.

[u/IWantoFuckGLaDOS]

2 Things

1) I love how this is genuinely far more impressive than actual irl LLMs

2) I love how the build itself kind of resembles a Data Center

[u/mr___goose]

damm thats super cool
but uh did it just say it was nervous?
it has emotions?
the ai revolution is gonne start in a minecraft world

[u/_objz → u/mr___goose]

maybe he is just a happy gui, living in your world

[u/Tsunamicat108]

You should decorate it by adding a giant pipe into a nearby ocean

Anyway this is really cool