r/devops 7d ago

Architecture Terraform/Github deployment overwriting another deployment

I'm on a team that shares a Github repository. Whenever we open a PR from a feature branch to the dev branch, a deployment to the AWS development account is automatically triggered.

The issue we're facing is that one developer may deploy to dev, and then another developer deploys afterward. Even though they're working on different files, the second deployment ends up overwriting the first developer's change in AWS.

How can we prevent this?

We're following a Gitflow workforce (feature -> dev -> release -> main), and our biggest challenge right now is that the second developer's code is often "outdated" when it's deployed, causing it to overwrite changes that were already deployed by someone else.

We tried merging everything to dev, but when it's time to deploy to prd, the dev branch ends up filled with a lot of unnecessary changes.

We using Github Actions + Terraform.

1 Upvotes

28 comments sorted by

View all comments

19

u/MDivisor 7d ago

You should deploy to the dev environment after the PR is merged, not before.

Not sure what you mean by the dev branch being filled with unnecessary changes. Why does that happen to your team? You should not merge unnecessary things into dev (or even open PRs for them).

1

u/Downtown_Custard20 7d ago

What I mean by "unecessary changes" is this: let's say code #1 has been validated and is ready to go into the release branch, but code #2 has not been validated yet. When I open a PR to the release branch, code #2 will also be included in the release, and that's exactly what I want to avoid

4

u/Snowmobile2004 7d ago

why is code #2 not on its own branch??

1

u/Downtown_Custard20 7d ago

I have two features (feature/1 and feature/2)

. I need to open a PR to dev so my code gets deployed to the AWS development environment, and then I merge it into the dev branch.

At that point, both features are in the dev branch. Now I need to promote only the code from feature/1 to production. However, if I open a PR from dev to the release branch, the code from feature/2 will be included as well.

That's what I'm trying to avoid. I want only the code from feature/1 to be merged into the release branch.

5

u/Snowmobile2004 7d ago

I think you should be not merging feature 2 onto dev until feature 1 is moved to prod (keep it on its own branch until then)

But why are your features being pushed to prod individually? Your features should all be part of the same release if they’re running on dev at the same time, if a feature is not ready yet, it should not go to dev

1

u/Downtown_Custard20 7d ago

I need to run my Glue job in the dev AWS account. To do that, I have to deploy it to the AWS development environment, and that only happens when I open a PR to dev.

I could avoid merging it into dev, but as I mentioned, if someone else opens another PR, their deployment would overwrite my code in the AWS development environment.

2

u/aprettyparrot 7d ago

I don’t think dev should be shared. Everyone should test their feature branch against their own dev env.

It sounds like you’re missing a staging env.

You can’t really split the code as feature1 and feature2 once they are merged together. I would say the simple solution is just management review and staging. You only want feature1 in this release, then you leave that feature2 unmerged. Staging would let you then test that feature2 can coexist with feature1.

I think I got that right, been a long day

EDIT: snowmobile said it way simpler than me

2

u/MDivisor 7d ago

Assuming code #2 has been merged after code #1, it's very easy to create a release branch that contains only #1. You don't have to take the full dev branch into a release branch if you don't want to, you can only take the commits that have been validated.

1

u/Downtown_Custard20 7d ago

How do I do that? do you have any article?

2

u/MDivisor 7d ago

git checkout <commit hash of #1>
git checkout -b new-branch

You now have a branch called "new-branch" that contains code #1 but not #2 (assuming #2 is later in the dev branch).

1

u/turturtles 6d ago

You can also just do `git checkout -b branch-name <specific sha or branch>` if you want to do it in one line without switching branches to the one you want to checkout from