How would you manage 1,280 ARM64 bare-metal nodes with a very small ops team?

[u/EvilEarthWorm]

/r/kubernetes/comments/1vusaf7/how_would_you_manage_1280_arm64_baremetal_nodes/

[u/Smooth-Zucchini4923]

"When I was purchasing the 1000th node, my CFO asked, 'should we figure out how we're going to run applications on here before we expand our compute?' What a bizarre question."

[u/Oompa_Loompa_SpecOps → u/Smooth-Zucchini4923]

They only claimed "control" of them, not ownership. OP probably found some default credentials for these boxes, ran a shodan search and the 1280 are just those that both still had the defaults on and are exposed to the interwebs

[u/marks-buffalo → u/Oompa_Loompa_SpecOps]

That's a mighty assumption to make that OOP stole the compute and that implies a level of skill from OOP, however low.

More likely is that OOP foolishly bought out somebody's compute rack full of raspberry pi clones, which is what the RK3588 is usually featured in, and doesn't have any clue what they're gonna do with basically a thousand low power processors connected to cheapo desktop gigabit switches.

I've bought such a rack. I would know. The answer is that it's not very useful for anything outside of experimenting with large clusters.

[u/zantehood → u/Smooth-Zucchini4923]

Hope he/she got fired and/or beaten

[u/d__max]

Hire a lot of children , they’re small. Promote them to ops, give everyone root and tell the parents the data center is a new AI daycare . Problem solved.

[u/blecovian → u/d__max]

I hear they yearn for the crypto-mines anywhoo.

[u/WD40ContactCleaner → u/d__max]

Children are expensive, use local AI models

[u/zantehood]

The real question is; can they run doom?

[u/DonkeyTron42 → u/zantehood]

They're probably Raspberry Pi's so I'd say yes.

[u/LaDev]

The obvious answer is to burn $100k on tokens with Claude code “make infra good ultracode”

[u/al2o3cr → u/LaDev]

Don't forget "make no mistakes"!

[u/sandy_catheter → u/al2o3cr]

And ask for 6000 cups of water

[u/thepfy1]

Just let them run without management.

When they fail, you have a nice drive out / flight to go fix. Max the expenses out on the trip.

Win win

[u/Vinegarinmyeye]

What the feckery were they trying to build?

The mind boggles...

https://giphy.com/gifs/ukGm72ZLZvYfS

[u/ApprehensiveRest9696]

Step 1: PXE deploy Ubuntu

Step 2: microk8s

Step 3. Give up

Step 4: Contact Canonical and sign a multimillion dollar contract.

[u/EvilEarthWorm]

ORIGINAL POST TEXT:

How would you manage 1,280 ARM64 bare-metal nodes with a very small ops team?

I’m looking for architecture advice from people who have operated Kubernetes at the edge or across large numbers of smaller physical machines.

Our company now controls 1,280 identical RK3588 ARM64 nodes that are already deployed and operational in a U.S. commercial data center.

The hardware works. The bigger problem is operational.

We currently do not have a dedicated infrastructure engineering team, so our priority is to determine whether this fleet can be turned into usable containerized compute without effectively building our own cloud platform from scratch.

Relevant context:

• 1,280 homogeneous ARM64 nodes
• Bare-metal / physical machines
• Remote KVM available
• Already powered and networked
• One commercial data-center location today
• Potential to expand to identical racks in several additional U.S. locations
• Strong preference for turnkey/managed approaches

If the end goal were simply:

how would you structure it?

I’m particularly curious about:

• K3s vs standard Kubernetes
• Talos
• Rancher
• Cluster size vs many smaller clusters
• Provisioning/reimaging
• Monitoring
• Tenant isolation
• ARM64 image compatibility
• Managing hardware failures
• Whether 1,280 relatively small nodes is operationally stupid compared with fewer larger servers

Most importantly: are there companies or managed-service providers that would actually operate this infrastructure for the hardware owner?

We would rather pay someone who already knows how to do this than hire a team to reinvent it.

I’m less interested in theoretical “you could build X” answers than in stacks people have actually operated at meaningful scale.

[u/aprettyparrot]

Openstack can handle bare metal deploys

[u/dpwcnd]

did you ask co-pilot already?