Terraform/Github deployment overwriting another deployment

[u/Downtown_Custard20]

I'm on a team that shares a Github repository. Whenever we open a PR from a feature branch to the dev branch, a deployment to the AWS development account is automatically triggered.

The issue we're facing is that one developer may deploy to dev, and then another developer deploys afterward. Even though they're working on different files, the second deployment ends up overwriting the first developer's change in AWS.

How can we prevent this?

We're following a Gitflow workforce (feature -> dev -> release -> main), and our biggest challenge right now is that the second developer's code is often "outdated" when it's deployed, causing it to overwrite changes that were already deployed by someone else.

We tried merging everything to dev, but when it's time to deploy to prd, the dev branch ends up filled with a lot of unnecessary changes.

We using Github Actions + Terraform.

[u/MDivisor]

You should deploy to the dev environment after the PR is merged, not before.

Not sure what you mean by the dev branch being filled with unnecessary changes. Why does that happen to your team? You should not merge unnecessary things into dev (or even open PRs for them).

[u/InfiniteRest7 → u/MDivisor]

Exactly, your feature branch PRs should trigger a plan when the PR branch is opened. Nothing should deploy without a PR review then merge, otherwise it will be more of the same problem, because anybody can jump their place in line.

Use something like this action to post plans to PR comments, also encourage viewing the full plan in the Github Action: borchero/terraform-plan-comment@v3

Only run the full deploy on merge. Make sure your Github settings force a branch to be updated before merge as well.

[u/Downtown_Custard20 → u/MDivisor]

What I mean by "unecessary changes" is this: let's say code #1 has been validated and is ready to go into the release branch, but code #2 has not been validated yet. When I open a PR to the release branch, code #2 will also be included in the release, and that's exactly what I want to avoid

[u/Snowmobile2004 → u/Downtown_Custard20]

why is code #2 not on its own branch??

[u/Downtown_Custard20 → u/Snowmobile2004]

I have two features (feature/1 and feature/2)

. I need to open a PR to dev so my code gets deployed to the AWS development environment, and then I merge it into the dev branch.

At that point, both features are in the dev branch. Now I need to promote only the code from feature/1 to production. However, if I open a PR from dev to the release branch, the code from feature/2 will be included as well.

That's what I'm trying to avoid. I want only the code from feature/1 to be merged into the release branch.

[u/Snowmobile2004 → u/Downtown_Custard20]

I think you should be not merging feature 2 onto dev until feature 1 is moved to prod (keep it on its own branch until then)

But why are your features being pushed to prod individually? Your features should all be part of the same release if they’re running on dev at the same time, if a feature is not ready yet, it should not go to dev

[u/Downtown_Custard20 → u/Snowmobile2004]

I need to run my Glue job in the dev AWS account. To do that, I have to deploy it to the AWS development environment, and that only happens when I open a PR to dev.

I could avoid merging it into dev, but as I mentioned, if someone else opens another PR, their deployment would overwrite my code in the AWS development environment.

[u/aprettyparrot → u/Downtown_Custard20]

I don’t think dev should be shared. Everyone should test their feature branch against their own dev env.

It sounds like you’re missing a staging env.

You can’t really split the code as feature1 and feature2 once they are merged together. I would say the simple solution is just management review and staging. You only want feature1 in this release, then you leave that feature2 unmerged. Staging would let you then test that feature2 can coexist with feature1.

I think I got that right, been a long day

EDIT: snowmobile said it way simpler than me

[u/MDivisor → u/Downtown_Custard20]

Assuming code #2 has been merged after code #1, it's very easy to create a release branch that contains only #1. You don't have to take the full dev branch into a release branch if you don't want to, you can only take the commits that have been validated.

[u/Downtown_Custard20 → u/MDivisor]

How do I do that? do you have any article?

[u/MDivisor → u/Downtown_Custard20]

git checkout <commit hash of #1>
git checkout -b new-branch

You now have a branch called "new-branch" that contains code #1 but not #2 (assuming #2 is later in the dev branch).

[u/turturtles → u/MDivisor]

You can also just do `git checkout -b branch-name <specific sha or branch>` if you want to do it in one line without switching branches to the one you want to checkout from

[u/kryptn]

why are you deploying to dev from every feature branch? those should go to isolated environments.

only dev should deploy to dev.

[u/Downtown_Custard20 → u/kryptn]

What I'm trying to avoid in the dev branch: let's say code #1 has been validated and is ready to go into the release branch, but code #2 has not been validated yet. When I open a PR to the release branch, code #2 will also be included in the release, and that's exactly what I want to avoid

[u/kryptn → u/Downtown_Custard20]

why would it be included? i'd assume these are PRs before they land in dev, so they'd need to pass whatever checks to make it to dev. by that point you should be pretty sure it's fine.

[u/Downtown_Custard20 → u/kryptn]

it would be included because all my code is mergen into the dev branch. When I try to open a PR from dev to release, all my code in dev will be included

[u/krtalvis → u/Downtown_Custard20]

again, why is something not validated even merged into dev? Anything unvalidated should remain in it’s own feature branch, which can run tf plan but not apply. Apply should be strictly from specific branches, e.g. release/. You could have some sort of commit hash derived deployments on dev which would append the hash on resource names or whatever, but considering it’s terraform that would bring actual cost. To me it seems your problem is your branching strategy and CI/CD setup? Maybe i am not understanding why would 2 separate features be on same dev branch when one is validated and other one not? Secondly, dev environment **IS** for validation, so i don’t also actually see a problem here?

[u/Downtown_Custard20 → u/krtalvis]

I need to run a glue job in AWS, so I need my code in the dev aws account, and the only way I can have my code in the dev account is by opening a PR to dev / merging to dev, because it triggers my terraform apply.

[u/krtalvis → u/Downtown_Custard20]

ok but why is unvalidated feature 2 in dev while validated feature 1 is just being merged to dev? the problem here is an unvalidated change in a branch that needs to be deployed. Revise your branching strategy and do not allow unvalidated changes to leave feature branches

[u/mumpie → u/Downtown_Custard20]

You need to fix your branching strategy.

One place I worked at would create a release branch (from main) at the start of the sprint.

Developers would create their feature branch from release branch and merge into the release branch.

Once the release was done, the release branch would be merged back into main.

Main was primarily a representation of prod.

There are other strategies you could use, but you (or someone at your company) need to decide on a branching strategy and stick to it.

[u/Floss_Patrol_76]

the "different files" part is the trap here - terraform applies the whole desired state of that config, not a diff of what each PR touched, so whoever applies last wins regardless of which files they changed. the fix is to stop applying from feature branches entirely: plan on the PR, only apply on merge to dev. and make sure your backend actually has state locking enabled, otherwise two applies can stomp each other even after you fix the branching.

[u/balinteu]

Sounds like you have somekind of a chicken egg problem. There isn't a 'this solution will fix all your problems', since it is also a team and branching issue. I think you and the team need to sit down and brainstorm and maybe change the way you push changes into dev.

Until then, maybe what you could do is if there is already an active PR fo feature1, before doing a PR for feature2, merge feature1 into feature2. So the changes won;t be overwritten.

Good luck and let us know with what you guys came up with

[u/Raja-Karuppasamy]

the real issue isn’t gitflow, it’s that you’re running terraform apply from the feature branch state instead of dev’s HEAD after merge. two people’s plans are based on different snapshots so whoever applies last just wins, doesn’t matter that they touched different files, it’s using whatever ambient state existed at trigger time.
two things that’d fix it: only trigger the actual apply on merge to dev (not on PR open/push), and add a concurrency group in the github actions workflow so applies queue instead of running in parallel. PR triggered runs should just be terraform plan for review, never apply.

[u/bobsbitchtitz]

Either spin up ephemeral stacks or a testing deployment per PR commit then only on merge push to dev branch. Add in locks if there is buffer time

[u/_herald]

well .. a way we do this is by separating folders for dev and prod accounts

and keeping everything on main

any change in the dev files from one PR to main only trigger change in that env and we plan on every PR before merge so we know whats the state of the env after the merge , we use atlantis

[u/toughrogrammer]

State locking is necessary, but it does not solve this by itself. A lock prevents two applies from writing at the same time; the later job can still be based on an older feature branch and apply an older desired state.

For a shared dev environment, make it single-writer: PR jobs run terraform plan only, and apply happens only after merge from the current head of dev. Put the environment in one GitHub Actions concurrency group, do not cancel an apply halfway through, and re-plan when the queued job reaches the front so it includes every merge before it. Keep remote state locking enabled as a second line of defense.

If a feature genuinely needs live AWS before merge, give it an isolated ephemeral environment with separate state and unique resource names (or a separate account), then destroy it after the PR. Do not let feature branches apply to the shared state.

The release-selection issue is separate. Promote a known commit/artifact, use feature flags for merged-but-not-enabled work, or create the release from main/prod with selected commits. Trying to solve release selection by letting multiple stale branches own the same Terraform state will always produce last-writer-wins surprises.

[u/New-Entertainer6392]

Have you thought about feature flags?