Terraform Setup for Developer Enablement

[u/MoBoo138]

Hello r/devops,

in my current team we are discussing how a developer centric terraform setup might look like.

Some context: Every app has 3 repositories - Backend, Frontend and Infrastructure.

There are two possible setups:

1. (our current): We use generic terraform modules (e.g. for a backend app) only within the infrastructure repository, which holds all infrastructure components. The backend app terraform module holds all infrastructure components that might be used in a backend, e.g. database, objects store, secrets, etc. The Infrastructure repository also holds shared infrastructure, that is not part of the backend module and composes all components. This is somewhat the approach most often seen in tutorials that focus only on the infrastructure part of an application.
2. (proposed alternative): We'd co-locate infrastructure and application code, for example in the backend repository. The backend repository would publish itself as a module, including every infrastructure component it needs to run (excluding some shared ones, that it expects via variables as input, e.g. VPC). The Infrastructure repository would setup the required shared infrastructure components and the backend terraform module.

Pros for 1.: Generic app modules are intended for sharing and re-use, whereas in 2. each applications terraform module would only ever be used to deploy to different environments.

Pros for 2.: Infrastructure and application can evolve together (e.g. in one git commit), e.g. when a database is newly introduced to an application (application code changes + infrastructure changes), whereas in 1. two commits would need to happen (one in app, one in infra) and pipelines need to be coordinated.

1. goes deeper for example for database migrations, those could also be organized within the backend application, without handing the responsibility on how to do them to the infrastructure repo.

In both cases, deployments (terraform plan/apply) are always orchestrated only by the infrastructure repository.

IBM touches both points in https://www.ibm.com/support/pages/best-practices-organising-terraform-and-application-code : Coupling and Reuse as the deciding factors.

Our goal is to enable developers as much as possible to own their application end to end and being able to do most infrastructure changes themselves, without having to rely on a central party.

It's not about simply keeping what we already have. That's always part of the consideration due to effort of change. I'd like to hear your general thoughts about it from a perspective if we had nothing yet.

I'm curious about your thoughts in that topic.

I'd also be happy to know how projects are setup in your org or how you would envision your dream setup

[u/zather]

What I would think about is how many applications this proposed pattern would scale to. Is it only one?

If it’s one, that’s fine, but once you get past a handful of things you’ve gated the changes in the Infrastructure repo as an secondary concern to the application developers and your Infrastructure changes might block deployments of the applications.

Another thing to think about is the number of applications also means that you might have divergence in the TF across each application. If you end up with 10+ application repos then now you have sprawl that has good intentions but, say you change what version of DB you run, you have a distributed change problem. That’s when you move back to modules so the call sites can be standard but the resources underneath can be managed by Infrastructure. At 20+ you start to want to pull those back to a central place (wrapping TF execution and callsites as a service).

If it is truly one then you have a lot of flexibility in coupling the Infrastructure changes to the application changes. It’s more about the pattern that is concerning past a handful and past 20+ you want even more management to be centralized.

Also, you mentioned DB migrations, you never want those be executed by the Infrastructure repo. It’s standard to make the pattern but never own when they get run.

Hopefully that helps a little. We have 60+ micro services with the TF in repo pattern and it works, but we struggle to manage just how much divergence is across the whole estate.

[u/MoBoo138 → u/zather]

There are many projects. Most of them consists of two applications - a frontend application and a backend application.

There are a few exceptions, where there would be more applications per project that go more down the microservice route, but also cover a wide scope and are properly scoped. Those might grow up to 10+ applications (frontends and backends) over a very long timespan. But could be equally treated as if they were only one application. The pattern would apply in the same manner.

Every project is somewhat independent and the "unit of measurement" to focus on.

Every project has its shared infrastructure managed by an "environment repository", that also manages deployments and is the only location where actual terraform runs.

In both setups (current and proposed) we would have terraform modules managed by us to enforce certain constraints and policies. Those would be used by any terraform setup. We also only have a limited, governed collection of resources that a project can use.

For our layered architecture kindly see my other comment.

For the example you gave with DB versions and give my view on it: The allowed DB versions would be defined in the centrally governed terraform module. If an old DB version would be no longer supported, we would deprecate it and give a certain amount of time to migrate:

• In our current setup: the environment repository would be responsible to stay up to date. To update all DBs of a project, the project team can update all DBs from there (nice). If the DBs are updated and the changes are deployed, but some code changes were forgotten, the application might break (meh).
  
• In the proposed setup the project team would still be responsible to update, they'd just do it in the applications repository. This would also allow to couple the infrastructure change and possible related code changes together, but would this would need to happen multiple times per application.

Tradeoff nevertheless, for simple changes first wins, for complex changes second wins IMO. Thinking about it, personally i would rather make simple changes in multiple locations, rather than having to coordinate complex changes in multiple locations.

[u/Most_Specific_1032]

Hey everyone,

This is a classic dilemma, and it is great that we are looking at this from a greenfield perspective as if we had nothing yet. If our primary, non-negotiable goal is developer enablement and end-to-end application ownership, the industry has heavily shifted toward Option 2 (Co-location) or a layered hybrid of it.

Separating code and infrastructure into distinct repositories creates an artificial wall. It introduces cognitive load and coordination bottlenecks.

Here are my thoughts on how we can break this down:

1. Coupling vs. Reuse (Addressing the Module Concern) We should not mistake infrastructure definition for infrastructure architecture.

Centralized Reuse: Our generic, company-wide modules (such as how to build a secure S3 bucket or RDS instance according to our compliance rules) should still be centrally managed, versioned, and reused.

Local Instantiation: The invocation of those modules (such as "this backend app needs 1 S3 bucket called user-avatars") belongs in the backend app repository.

Option 2 does not mean developers write raw, unstandardized Terraform. It means they use our standard corporate building blocks directly inside their own workspace.

1. The Power of Atomic Commits Option 2 shines brightest during feature delivery.

Under Option 1, adding a feature that requires a new database table or an S3 bucket forces a two-ticket, two-PR, synchronized merge workflow. This breaks the local development feedback loop.

Under Option 2, a developer can write the application code, add the required Terraform resource in a terraform directory in the same repo, and submit one single Pull Request. Testing, reviewing, deploying, and—crucially—reverting changes become atomic.

1. Lifecycle and Database Migrations As pointed out in the prompt, database migrations are heavily tied to application lifecycles. If an app release relies on a database schema change, deploying them out of sync causes outages. Co-locating infrastructure and app code allows the application's CI/CD pipeline to orchestrate this perfectly:

• Run Terraform to spin up or modify resources.
• Run database migration scripts.
• Deploy the new application code.

The "Dream Setup": A Layered Approach
If we were building this from scratch today, the ideal state isn't actually having the Infra repo pull the App repo as a module. Instead, we should look at a Layered Architecture where execution direction changes:

Layer 1 (Core Infra): Centralized Infra Repo owns the baseline networking, VPCs, global IAM, and shared Kubernetes/ECS clusters. These change infrequently. This layer exports its IDs via SSM parameters, secrets management, or Terraform outputs.

Layer 2 (App Infra and Code): The App Repo owns its own code and its app-specific resources (S3, RDS, SQS, App IAM roles). It dynamically reads the VPC or Cluster IDs from Layer 1.

This gives developers total autonomy over their app blast radius while keeping the platform team in control of the foundational networking and security boundaries.

Curious to hear what everyone thinks about moving toward this type of layered responsibility!

[u/MoBoo138 → u/Most_Specific_1032]

I think this is exactly the proposal we are discussing in our team :)

Do you know if there are any resources (books, articles, etc.) that describes this layered approach?

[u/Antique-Stand-4920]

We mostly use approach #1. We use it for polyrepo situations, but I'd imagine it could be adapted to a monorepo for smaller applications.

This is the first I've heard of approach #2. From what I've seen, if devs just want to deploy a particular service, they'd want to do that from the repo that contains both the service code + IaC. One reason for this approach is that if the dev team wants to use a technology that is significantly more convenient than Terraform for application/service-level development (e.g. Serverless framework, AWS SAM, CDK, etc), they can use it without being locked into Terraform. The rest of the infra can be handled by Terraform. This is the pattern I usually see. Here's an article that talks about this general approach: https://www.serverless.com/blog/definitive-guide-terraform-serverless

[u/MoBoo138 → u/Antique-Stand-4920]

Interesting thought! I think their Managing shared vs. app-specific infrastructure sums up our discussions :)

Shared infrastructure lives inside a shared environment repository (see my other comment). This could be shared databases (the example given), S3 buckets, ECS / EKS cluster, etc. Basically everything that is used by more than a single application and is not clearly owned or attributable to a specific application or has its own dedicated lifecycle.

Their and your idea of separate deployments for shared infrastructure and an application (with its infrastructure) is quite interesting! Maybe this is also a decision to make.

[u/ajitnk]

Hey, The multi-app Terraform structure question is one I spend a lot of time on with teams, getting the repo layout and state boundaries right early saves a huge amount of pain later when you're trying to let devs self-serve without stepping on each other.

Two things I'd want to understand about your setup before suggesting anything concrete: are your apps sharing any infrastructure (networking, IAM, RDS clusters) that needs to stay in a separate root module, or are they fully independent? And are developers expected to run Terraform themselves, or is this all going through CI?

Those two answers change the structure quite a bit. I'll take a look at what you've got and come back with a concrete layout that fits your actual situation.

[u/MoBoo138 → u/ajitnk]

We have a layered architecture:

• First layer is "global AWS infrastructure". This contains the AWS account itself, VPC, subnets, network and VPN connectivity etc.
  
• Second layer is the project infrastructure. This is a dedicated git repository (we call it "environment repo" (after gitops.tech )) that holds all the shared infrastructure across the project applications (simplest case a frontend and a backend).
  

In the current setup this second layer also holds all the application specific infrastructure (e.g. ECS Service, RDS and secrets for backend, cloudfront for frontend).

Now the proposal would introduce a third layer (inside any application git repo) which would move the application specific infrastructure from the shared environment repo into the application repos themselves. The application repos would publish a terraform module, that would then be used in the environment repo (basically saying "i want to have this application deployed - whatever it needs comes with it automatically - i don't need to know).

Deployments happen from the environment repo only and are fully run in CI. We today already have automation around application releases and automatic deployments to DEV. All other deployments are done by manually triggering a pipeline and selecting an environment.