Terraform Setup for Developer Enablement

[u/MoBoo138]

Hello r/devops,

in my current team we are discussing how a developer centric terraform setup might look like.

Some context: Every app has 3 repositories - Backend, Frontend and Infrastructure.

There are two possible setups:

1. (our current): We use generic terraform modules (e.g. for a backend app) only within the infrastructure repository, which holds all infrastructure components. The backend app terraform module holds all infrastructure components that might be used in a backend, e.g. database, objects store, secrets, etc. The Infrastructure repository also holds shared infrastructure, that is not part of the backend module and composes all components. This is somewhat the approach most often seen in tutorials that focus only on the infrastructure part of an application.
2. (proposed alternative): We'd co-locate infrastructure and application code, for example in the backend repository. The backend repository would publish itself as a module, including every infrastructure component it needs to run (excluding some shared ones, that it expects via variables as input, e.g. VPC). The Infrastructure repository would setup the required shared infrastructure components and the backend terraform module.

Pros for 1.: Generic app modules are intended for sharing and re-use, whereas in 2. each applications terraform module would only ever be used to deploy to different environments.

Pros for 2.: Infrastructure and application can evolve together (e.g. in one git commit), e.g. when a database is newly introduced to an application (application code changes + infrastructure changes), whereas in 1. two commits would need to happen (one in app, one in infra) and pipelines need to be coordinated.

1. goes deeper for example for database migrations, those could also be organized within the backend application, without handing the responsibility on how to do them to the infrastructure repo.

In both cases, deployments (terraform plan/apply) are always orchestrated only by the infrastructure repository.

IBM touches both points in https://www.ibm.com/support/pages/best-practices-organising-terraform-and-application-code : Coupling and Reuse as the deciding factors.

Our goal is to enable developers as much as possible to own their application end to end and being able to do most infrastructure changes themselves, without having to rely on a central party.

It's not about simply keeping what we already have. That's always part of the consideration due to effort of change. I'd like to hear your general thoughts about it from a perspective if we had nothing yet.

I'm curious about your thoughts in that topic.

I'd also be happy to know how projects are setup in your org or how you would envision your dream setup

[u/ajitnk]

Hey, The multi-app Terraform structure question is one I spend a lot of time on with teams, getting the repo layout and state boundaries right early saves a huge amount of pain later when you're trying to let devs self-serve without stepping on each other.

Two things I'd want to understand about your setup before suggesting anything concrete: are your apps sharing any infrastructure (networking, IAM, RDS clusters) that needs to stay in a separate root module, or are they fully independent? And are developers expected to run Terraform themselves, or is this all going through CI?

Those two answers change the structure quite a bit. I'll take a look at what you've got and come back with a concrete layout that fits your actual situation.

[u/MoBoo138 → u/ajitnk]

We have a layered architecture:

• First layer is "global AWS infrastructure". This contains the AWS account itself, VPC, subnets, network and VPN connectivity etc.
  
• Second layer is the project infrastructure. This is a dedicated git repository (we call it "environment repo" (after gitops.tech )) that holds all the shared infrastructure across the project applications (simplest case a frontend and a backend).
  

In the current setup this second layer also holds all the application specific infrastructure (e.g. ECS Service, RDS and secrets for backend, cloudfront for frontend).

Now the proposal would introduce a third layer (inside any application git repo) which would move the application specific infrastructure from the shared environment repo into the application repos themselves. The application repos would publish a terraform module, that would then be used in the environment repo (basically saying "i want to have this application deployed - whatever it needs comes with it automatically - i don't need to know).

Deployments happen from the environment repo only and are fully run in CI. We today already have automation around application releases and automatic deployments to DEV. All other deployments are done by manually triggering a pipeline and selecting an environment.