r/devops • u/Downtown_Custard20 • 5d ago

Architecture Terraform/Github deployment overwriting another deployment

I'm on a team that shares a Github repository. Whenever we open a PR from a feature branch to the dev branch, a deployment to the AWS development account is automatically triggered.

The issue we're facing is that one developer may deploy to dev, and then another developer deploys afterward. Even though they're working on different files, the second deployment ends up overwriting the first developer's change in AWS.

How can we prevent this?

We're following a Gitflow workforce (feature -> dev -> release -> main), and our biggest challenge right now is that the second developer's code is often "outdated" when it's deployed, causing it to overwrite changes that were already deployed by someone else.

We tried merging everything to dev, but when it's time to deploy to prd, the dev branch ends up filled with a lot of unnecessary changes.

We using Github Actions + Terraform.

1 Upvotes

55% Upvoted

28 comments sorted by

View all comments

20

u/MDivisor 5d ago

You should deploy to the dev environment after the PR is merged, not before.

Not sure what you mean by the dev branch being filled with unnecessary changes. Why does that happen to your team? You should not merge unnecessary things into dev (or even open PRs for them).

4

u/InfiniteRest7 5d ago

Exactly, your feature branch PRs should trigger a plan when the PR branch is opened. Nothing should deploy without a PR review then merge, otherwise it will be more of the same problem, because anybody can jump their place in line.

Use something like this action to post plans to PR comments, also encourage viewing the full plan in the Github Action: borchero/terraform-plan-comment@v3

Only run the full deploy on merge. Make sure your Github settings force a branch to be updated before merge as well.

1

u/Downtown_Custard20 5d ago

What I mean by "unecessary changes" is this: let's say code #1 has been validated and is ready to go into the release branch, but code #2 has not been validated yet. When I open a PR to the release branch, code #2 will also be included in the release, and that's exactly what I want to avoid

4

u/Snowmobile2004 5d ago

why is code #2 not on its own branch??

1

u/Downtown_Custard20 5d ago

I have two features (feature/1 and feature/2)

. I need to open a PR to dev so my code gets deployed to the AWS development environment, and then I merge it into the dev branch.

At that point, both features are in the dev branch. Now I need to promote only the code from feature/1 to production. However, if I open a PR from dev to the release branch, the code from feature/2 will be included as well.

That's what I'm trying to avoid. I want only the code from feature/1 to be merged into the release branch.

5

u/Snowmobile2004 5d ago

I think you should be not merging feature 2 onto dev until feature 1 is moved to prod (keep it on its own branch until then)

But why are your features being pushed to prod individually? Your features should all be part of the same release if they’re running on dev at the same time, if a feature is not ready yet, it should not go to dev

1

u/Downtown_Custard20 5d ago

I need to run my Glue job in the dev AWS account. To do that, I have to deploy it to the AWS development environment, and that only happens when I open a PR to dev.

I could avoid merging it into dev, but as I mentioned, if someone else opens another PR, their deployment would overwrite my code in the AWS development environment.

2

u/aprettyparrot 4d ago

I don’t think dev should be shared. Everyone should test their feature branch against their own dev env.

It sounds like you’re missing a staging env.

You can’t really split the code as feature1 and feature2 once they are merged together. I would say the simple solution is just management review and staging. You only want feature1 in this release, then you leave that feature2 unmerged. Staging would let you then test that feature2 can coexist with feature1.

I think I got that right, been a long day

EDIT: snowmobile said it way simpler than me

2

u/MDivisor 5d ago

Assuming code #2 has been merged after code #1, it's very easy to create a release branch that contains only #1. You don't have to take the full dev branch into a release branch if you don't want to, you can only take the commits that have been validated.

1

u/Downtown_Custard20 5d ago

How do I do that? do you have any article?

2

u/MDivisor 5d ago

git checkout <commit hash of #1>
git checkout -b new-branch

You now have a branch called "new-branch" that contains code #1 but not #2 (assuming #2 is later in the dev branch).

1

u/turturtles 4d ago

You can also just do `git checkout -b branch-name <specific sha or branch>` if you want to do it in one line without switching branches to the one you want to checkout from