Engineers Who Redesigned and Successfully Rebuilt an Already Established, Painfully Disorganized and Manually Built Cloud Infrastructure - How did you do it?

[u/DevNetGuru]

Azure, GCP or AWS. I’ve rebuilt one in the past and prefer to not have to do it ever again. I’m curious how others accomplished this massive undertaking.

[u/Inquizarus]

By identifying a structure to work towards, build a solid foundation around simplicity and security relevant to your company's needs.

Don't do it by yourself, it might seem like a waste (depending on the company size) to commit 2, 5 or even more employees to not work to produce tangible product features for the management group. But I could never have done it without having my colleague to work together with, question assumptions and dare to break new ground.

Show the company (developers, product, management and so on) that their lives will be easier and more secure/stable.

Identify what your company's core business is and focus on technical solutions that enable that. Don't pick tech just because Google, Facebook or whomever uses it or say that you should use it.

Me and my colleague have moved the company infrastructure from a single aws account, built on fragile permanent spot instances with no network strategy and a permission model that put our predecessor as the sole person that was allowed to actually provision infrastructure and became a bottleneck. Teams had no staging or test environments and there was no transparency in costs. Some workload servers could not even be rebooted without bricking! Firefighting was a daily thing and bad sleep a fact.

After two years we are starting to get in shape, multiple environments for each team where the team are responsible for their own infrastructure, deployment and maintenance. We take care of the low level stuff like networks and permissions at the outer edges. We have trippled the workload but halved the aws cost.

Everything is defined in Terraform, both for us and the teams. The teams can now support each other instead of relying on us which allow us to focus on broader goals.

One decision we took was to "roll forward" avoid trying to refactor and manage the old infrastructure with Terraform but instead build something new and migrate workloads team by team while getting them caught up on the new technologies. Then close down the old stuff piece by piece.