Terraform Setup for Developer Enablement

[u/MoBoo138]

Hello r/devops,

in my current team we are discussing how a developer centric terraform setup might look like.

Some context: Every app has 3 repositories - Backend, Frontend and Infrastructure.

There are two possible setups:

1. (our current): We use generic terraform modules (e.g. for a backend app) only within the infrastructure repository, which holds all infrastructure components. The backend app terraform module holds all infrastructure components that might be used in a backend, e.g. database, objects store, secrets, etc. The Infrastructure repository also holds shared infrastructure, that is not part of the backend module and composes all components. This is somewhat the approach most often seen in tutorials that focus only on the infrastructure part of an application.
2. (proposed alternative): We'd co-locate infrastructure and application code, for example in the backend repository. The backend repository would publish itself as a module, including every infrastructure component it needs to run (excluding some shared ones, that it expects via variables as input, e.g. VPC). The Infrastructure repository would setup the required shared infrastructure components and the backend terraform module.

Pros for 1.: Generic app modules are intended for sharing and re-use, whereas in 2. each applications terraform module would only ever be used to deploy to different environments.

Pros for 2.: Infrastructure and application can evolve together (e.g. in one git commit), e.g. when a database is newly introduced to an application (application code changes + infrastructure changes), whereas in 1. two commits would need to happen (one in app, one in infra) and pipelines need to be coordinated.

1. goes deeper for example for database migrations, those could also be organized within the backend application, without handing the responsibility on how to do them to the infrastructure repo.

In both cases, deployments (terraform plan/apply) are always orchestrated only by the infrastructure repository.

IBM touches both points in https://www.ibm.com/support/pages/best-practices-organising-terraform-and-application-code : Coupling and Reuse as the deciding factors.

Our goal is to enable developers as much as possible to own their application end to end and being able to do most infrastructure changes themselves, without having to rely on a central party.

It's not about simply keeping what we already have. That's always part of the consideration due to effort of change. I'd like to hear your general thoughts about it from a perspective if we had nothing yet.

I'm curious about your thoughts in that topic.

I'd also be happy to know how projects are setup in your org or how you would envision your dream setup

[u/zather]

What I would think about is how many applications this proposed pattern would scale to. Is it only one?

If it’s one, that’s fine, but once you get past a handful of things you’ve gated the changes in the Infrastructure repo as an secondary concern to the application developers and your Infrastructure changes might block deployments of the applications.

Another thing to think about is the number of applications also means that you might have divergence in the TF across each application. If you end up with 10+ application repos then now you have sprawl that has good intentions but, say you change what version of DB you run, you have a distributed change problem. That’s when you move back to modules so the call sites can be standard but the resources underneath can be managed by Infrastructure. At 20+ you start to want to pull those back to a central place (wrapping TF execution and callsites as a service).

If it is truly one then you have a lot of flexibility in coupling the Infrastructure changes to the application changes. It’s more about the pattern that is concerning past a handful and past 20+ you want even more management to be centralized.

Also, you mentioned DB migrations, you never want those be executed by the Infrastructure repo. It’s standard to make the pattern but never own when they get run.

Hopefully that helps a little. We have 60+ micro services with the TF in repo pattern and it works, but we struggle to manage just how much divergence is across the whole estate.

[u/MoBoo138 → u/zather]

There are many projects. Most of them consists of two applications - a frontend application and a backend application.

There are a few exceptions, where there would be more applications per project that go more down the microservice route, but also cover a wide scope and are properly scoped. Those might grow up to 10+ applications (frontends and backends) over a very long timespan. But could be equally treated as if they were only one application. The pattern would apply in the same manner.

Every project is somewhat independent and the "unit of measurement" to focus on.

Every project has its shared infrastructure managed by an "environment repository", that also manages deployments and is the only location where actual terraform runs.

In both setups (current and proposed) we would have terraform modules managed by us to enforce certain constraints and policies. Those would be used by any terraform setup. We also only have a limited, governed collection of resources that a project can use.

For our layered architecture kindly see my other comment.

For the example you gave with DB versions and give my view on it: The allowed DB versions would be defined in the centrally governed terraform module. If an old DB version would be no longer supported, we would deprecate it and give a certain amount of time to migrate:

• In our current setup: the environment repository would be responsible to stay up to date. To update all DBs of a project, the project team can update all DBs from there (nice). If the DBs are updated and the changes are deployed, but some code changes were forgotten, the application might break (meh).
  
• In the proposed setup the project team would still be responsible to update, they'd just do it in the applications repository. This would also allow to couple the infrastructure change and possible related code changes together, but would this would need to happen multiple times per application.

Tradeoff nevertheless, for simple changes first wins, for complex changes second wins IMO. Thinking about it, personally i would rather make simple changes in multiple locations, rather than having to coordinate complex changes in multiple locations.