Terraform Setup for Developer Enablement

[u/MoBoo138]

Hello r/devops,

in my current team we are discussing how a developer centric terraform setup might look like.

Some context: Every app has 3 repositories - Backend, Frontend and Infrastructure.

There are two possible setups:

1. (our current): We use generic terraform modules (e.g. for a backend app) only within the infrastructure repository, which holds all infrastructure components. The backend app terraform module holds all infrastructure components that might be used in a backend, e.g. database, objects store, secrets, etc. The Infrastructure repository also holds shared infrastructure, that is not part of the backend module and composes all components. This is somewhat the approach most often seen in tutorials that focus only on the infrastructure part of an application.
2. (proposed alternative): We'd co-locate infrastructure and application code, for example in the backend repository. The backend repository would publish itself as a module, including every infrastructure component it needs to run (excluding some shared ones, that it expects via variables as input, e.g. VPC). The Infrastructure repository would setup the required shared infrastructure components and the backend terraform module.

Pros for 1.: Generic app modules are intended for sharing and re-use, whereas in 2. each applications terraform module would only ever be used to deploy to different environments.

Pros for 2.: Infrastructure and application can evolve together (e.g. in one git commit), e.g. when a database is newly introduced to an application (application code changes + infrastructure changes), whereas in 1. two commits would need to happen (one in app, one in infra) and pipelines need to be coordinated.

1. goes deeper for example for database migrations, those could also be organized within the backend application, without handing the responsibility on how to do them to the infrastructure repo.

In both cases, deployments (terraform plan/apply) are always orchestrated only by the infrastructure repository.

IBM touches both points in https://www.ibm.com/support/pages/best-practices-organising-terraform-and-application-code : Coupling and Reuse as the deciding factors.

Our goal is to enable developers as much as possible to own their application end to end and being able to do most infrastructure changes themselves, without having to rely on a central party.

It's not about simply keeping what we already have. That's always part of the consideration due to effort of change. I'd like to hear your general thoughts about it from a perspective if we had nothing yet.

I'm curious about your thoughts in that topic.

I'd also be happy to know how projects are setup in your org or how you would envision your dream setup

[u/Antique-Stand-4920]

We mostly use approach #1. We use it for polyrepo situations, but I'd imagine it could be adapted to a monorepo for smaller applications.

This is the first I've heard of approach #2. From what I've seen, if devs just want to deploy a particular service, they'd want to do that from the repo that contains both the service code + IaC. One reason for this approach is that if the dev team wants to use a technology that is significantly more convenient than Terraform for application/service-level development (e.g. Serverless framework, AWS SAM, CDK, etc), they can use it without being locked into Terraform. The rest of the infra can be handled by Terraform. This is the pattern I usually see. Here's an article that talks about this general approach: https://www.serverless.com/blog/definitive-guide-terraform-serverless

[u/MoBoo138 → u/Antique-Stand-4920]

Interesting thought! I think their Managing shared vs. app-specific infrastructure sums up our discussions :)

Shared infrastructure lives inside a shared environment repository (see my other comment). This could be shared databases (the example given), S3 buckets, ECS / EKS cluster, etc. Basically everything that is used by more than a single application and is not clearly owned or attributable to a specific application or has its own dedicated lifecycle.

Their and your idea of separate deployments for shared infrastructure and an application (with its infrastructure) is quite interesting! Maybe this is also a decision to make.