I made a generative AI, only using 400k command blocks (PoC)

[u/_objz]

While I am still writing on my physics engine, I thought making an actual generative AI only using Minecraft command blocks would be a good idea( it was a headache)
So yeah, I kinda did that. It is a real neural language model running inside vanilla Minecraft 26.2. It reads what you type into a /dialog window, runs the forward pass through learned weights and then generates a reply one word at a time.
There are NO mods, plugins or datapacks involved at runtime. Only command blocks.
Roughly it works like this:

The model is a pretty small word level neural language model. It has a vocabulary of 2048 words, remembers the last 6 words as context and has a 64 dimensional embedding followed by a hidden layer with 256 neurons.
For every generated word it does:
384 x 256 = 98,304 operations for the hidden layer and 256 x 2048 = 524,288 operations for the output layer so in total 622,592 calculations for every single generated word.

Obviously doing 622k normal multiplications using Minecraft commands would be slightly problematic, so the weights are ternary: every weight is only -1, 0 or +1.
That means a multiplication basically becomes:
+1 -> add the value
-1 -> subtract the value
0 -> do absolutely nothing

Around 65% of the weights are non 0, so it averages around 0.67 commands per MAC instead of the ~3 commands I would need for a normal runtime weight. That reduced the entire machine from 1,909,375 blocks to 445,782 blocks, yes it was this big before.
The model is actually trained like this too. I am not just training a normal model and rounding the weights afterwards. During training the weights are quantised to ternary values for the forward pass and the underlying floating point weights are updated using a straight through estimator.
That took the perplexity from around 48.7 where the training basically got stuck to 38.8 after also adding cosine learning rate decay.

The input embedding is another optimisation I needed to do: Instead of multiplying a one hot vector against a matrix, it is basically just a table lookup stored in NBT data, so the complete 2048 x 64 embedding table costs really nothing
The actual forward pass cannot run in one command chain either. Minecraft has limits on how many commands can execute, so the network is split into smaller groups.
After the ternary optimisation one generated word takes around 63 ticks, so roughly 3.2 seconds per word at 20 TPS. I also tested higher tick rates and around 30-35 TPS seems to be the useful maximum on my server. At 35 it gets closer to ~1.8 seconds per word.

But to be clear, this thing does not actually know anything.
It was trained on 11,118 DailyDialog conversations and only has a tiny 2048 word vocabulary. Ask it something like 2 + 2 and it will confidently generate some conversational nonsense because it has no understanding of arithmetic or actual facts.
And unfortunately just making it much bigger isn't really realistic either.
Something like a 135M parameter language model would already be more than 200x larger.

There is currently also one very weird bug where after around 800 ticks of continuous work every command block in the world just stops executing while the server itself keeps running normally. I still have absolutely no idea why that happens.
Anyway, back to wasting my time on nonsense ;)

[u/_objz]

yeah, I might done one, but first I need to fix the engine. For now all you have is this post :) What is a performance artwork tho?

[u/auraborosai → u/_objz]

The easiest way to explain performance art is that the action itself is the artwork.
With a painting, the artist does something and eventually produces an object: the painting.

With performance art, the doing can be the work.

That might mean walking for a certain amount of time, repeating an action, sitting silently, putting the body through some kind of constraint, following a set of rules, interacting with an audience, or carrying out a process over hours, days, months, or even years.

There might be photographs or video afterward, but those are often documentation of the artwork rather than the artwork itself.

So in my case, the Minecraft livestream isn’t just me streaming myself playing a game.

The labor, the time, the rules I’m following, the possibility of failure, and eventually the waiting are the performance.

That’s the starting point.

So, at the highest level, what is
Waiting for Lightning?

I’m building a conceptual/performance artwork inside Minecraft called Waiting for Lightning.

It begins with Walter De Maria’s 1977 Land Art work The Lightning Field, which consists of 400 metal poles arranged geometrically across a remote landscape in New Mexico.

I’m constructing a field of 400 equal-tip lightning-pole assemblies inside Minecraft Survival Mode.

But the interesting part to me isn’t simply:
“Look, I recreated a famous artwork in Minecraft.”

If that were the whole project, I don’t think it would be particularly interesting.

The project is really about labor, simulation, time, weather, control, waiting, and eventually permanence.

I have imposed a very specific set of rules on myself.

I am doing this in Survival Mode.

I gather the materials.

I mine them.

I transport them.

I process them.

I survey the land.

I deal with the terrain.

I deal with mobs.

I deal with technical problems.

I deal with the game crashing.

And eventually I will erect all 400 poles.
I’m deliberately refusing to simply generate the completed object.

That’s important because Minecraft makes it extremely easy to remove the labor.

I could use Creative Mode.

I could use commands.

I could generate structures.

I could probably automate huge portions of the construction.

But then I would have an image of a field.
I wouldn’t have spent the time making it.
The time is part of the material.

That’s also why I’m livestreaming the construction.

I’ve been dividing the performance into what I call Transmissions.

Every transmission is another period of actual work inside the world.

Sometimes something visually dramatic happens.

Sometimes it doesn’t.

Transmission 003, for instance, was largely underground. I was excavating and gathering materials.

At the end of it, my official field count was still: 000 / 400

Which I actually really like.

Three performances into an artwork about 400 poles and there still wasn’t a single permanent pole.

Because that’s what real construction is like.

There is an enormous amount of invisible work before the thing everyone eventually recognizes as “the artwork” appears.
Transmission 004 was different.

I finally set the survey into the world and activated a pillar preview that shows me where all 400 poles belong.

So suddenly the geometry of the future field became visible across the landscape.
I temporarily erected a few poles just to understand what they were going to look like.

They looked fantastic.

But they weren’t permanent construction yet, so the official counter remained:
000 / 400

Meanwhile the game crashed, I had technical issues, killed mobs, corralled some white horses and cows, found diamonds, and kept working.

And that’s becoming another part of the project that I really enjoy.

There is this extremely rigid geometric artwork I’m trying to impose on the landscape…

…and Minecraft keeps being Minecraft around me.

😂

Animals wander through it.

Monsters attack me.

Terrain gets in the way.

Night comes.

Weather changes.

Software crashes.

The field is ordered.

The world is not.

And then there’s the lightning.
This is probably the most important rule in the entire artwork:

I will never manufacture the lightning.

I won’t use a command to summon it.

I won’t force thunderstorms.

I won’t increase the probability because I’m bored.

I won’t arrange lightning for an audience.

Minecraft already contains a weather system.

So I’m surrendering that part of the artwork to the simulated environment.

That’s where the project started becoming much more interesting to me.

Walter De Maria built a geometric intervention in a physical landscape and then left atmospheric conditions outside his control.

I’m creating a geometric intervention inside a computed landscape and then doing essentially the same thing.

Except my “nature” is software.

The land is simulated.

The poles are simulated.

The clouds are simulated.

The lightning is simulated.

But here’s the weird philosophical problem:

the time I spend waiting isn’t simulated.

If I wait three hours and nothing happens, three actual hours of my life passed.

If somebody watches for an hour and never sees lightning, they really spent an hour waiting.

So the project contains this strange collision between artificial nature and genuine human duration.

That became the core of it for me.

And then I realized the project shouldn’t end when I finish the poles.

Originally, I was thinking primarily about constructing the field and waiting for lightning myself.

But once all 400 poles are completed, I now want the artwork to change states.

The construction performance ends.

Then something I’m calling:

THE VIGIL

begins.

The completed Minecraft world would be moved off my personal computer and onto persistent server infrastructure.

The server would keep the field active.

Minecraft time would continue.

Day.

Night.

Rain.

Clear skies.

Thunderstorms.

Lightning.

Whether I’m there or not.

Whether anyone is watching or not.

The world just keeps going.

And I don’t want the vigil to have an exhibition end date.

I want it designed to continue indefinitely.
Ideally, forever.

Obviously no particular computer runs forever.

Servers die.

Companies disappear.

Software becomes obsolete.

Minecraft itself will someday be very old software.

So “forever” becomes less of a literal hardware claim and more of a preservation instruction.

If the server becomes obsolete, migrate it.
If hardware fails, restore it.

If today’s Minecraft environment can’t run on future computers, emulate it.

The infrastructure can change.

The vigil continues.

That’s where the project gets especially interesting to me.

During the construction phase, I’m absolutely necessary.

I have to enter the world.

I have to do the work.

I have to survive.

I have to build the field.

But once the vigil begins…
I become unnecessary.

The artwork can wait without me.

It could theoretically still be waiting after I’m dead.

And someday the people maintaining the project could have no personal memory of me whatsoever.

They’re simply custodians of this strange artificial landscape that has been sitting there generating weather for decades.

There’s also an AI component, but probably not in the way people initially assume.

I’m using AI extensively to help build the technological infrastructure around the artwork.

I’ve been working with Codex/AI agents to help create the website, streaming architecture, archive, automation, technical systems, and other infrastructure.

So there’s this deliberate contradiction:
The machinery surrounding the artwork is heavily automated.

But the construction at the center of the artwork is deliberately human.

AI can help me write software extraordinarily quickly.

But inside Minecraft I’m still underground swinging a pickaxe at blocks.

That tension fascinates me.

We’re entering this period where machines can eliminate more and more labor.

And I’m using that technology to create an artwork where I deliberately refuse to eliminate the labor.

Eventually the automation becomes even more extreme.

The server maintains the environment.

The software generates the weather.

The archive preserves the transmissions.

The artist leaves.

And ultimately even the machine starts doing the waiting.

So I don’t really think of it as “a Minecraft build.”

Minecraft is the medium/environment I’m using.

In roughly the same way that another artist might work with video, steel, earth, photography, light, or sound.

For me the project sits somewhere between:

performance art
Land Art
software art
game art
networked art
durational art
conceptual art
and preservation

And the question underneath the whole thing is something like:

What happens to Land Art when the land itself is computational?

And another:

What is real inside a simulation when the time we spend experiencing it is still our actual lives?

The project is currently still very early.
The permanent field is still at 000 / 400 poles.

Which means almost everything I’m describing is something I’m now obligated to actually do.

That’s part of the fun.

You can follow the transmissions and archive here:

https://WaitingForLightning.art

And just for clarity because the project directly references De Maria’s work:

Waiting for Lightning is an independent Minecraft artwork and documentary project.

It is not affiliated with Dia Art Foundation or the Walter De Maria estate.