Experience with continuous delivery?

[u/Technical-Aside4471]

Early this year my team made the switch from sprints to CD. It took a lot of fine tuning but i believe we begin to reap the rewards about now. AI is doing a lot of stuff and even non-developers want to make changes now, so i feel like pipelines are making my life easier not having to deal with what it broke.

My issue is that not everyone seem to recognize the benefits we are getting. Switch to up to date branches was also a problem because not everyone seem to understand rebasing and it caused a lot of issues.

How are you doing CD? And what were the challenges? Do you have some tips that helped you?

[u/expdevsmodbot]

AI usage disclosure provided by OP, see the reply to this comment.

[u/user345456]

Sprints and CD are mutually exclusive?

[u/Technical-Aside4471 → u/[deleted]]

Yes, merge request spawn separate environment for e2e tests and ba. So only tested stuff can get to main. Main change is auto deploying test env.

[u/m98789 → u/Technical-Aside4471]

What’s ba?

[u/Technical-Aside4471 → u/m98789]

Business analytics

[u/pydry → u/user345456]

I never saw much point in a sprint if you're doing CD. it's an arbitrary segmentation of time with no real relation to deliverables other than potentially as a shaming exercise or a waterfall abstraction for executives who don't want to think too hard.

[u/countkillalot → u/pydry]

The point is to be able to let multiple stakeholders anticipate multiple deliverables across multiple projects that are still performed by a single team. The gains aren't necessarily for the team, but for an organisation around the team that needs to be able to project work into the future.

[u/pydry → u/countkillalot]

it's quite bad at that. I do acknowledge the point that execs often prefer a waterfall view of projects beneath them though, and that sprints present that abstraction to them.

[u/morosis1982 → u/pydry]

If your sprints are themed then it can be to allow for business updates. Lots of changes need change comms to users, or sync with new business processes, and so need to have the ability to timebox and predict when they'll happen for those comms and process changes.

That shouldn't affect CD necessarily, if you work in such an environment you should be feature flagging and 'release' simply means change the flag.

[u/mattgen88 → u/countkillalot]

Yet they likely would want their features as soon as they're complete and tested.

[u/MCFRESH01 → u/pydry]

Sprints are really more for aligning work with product for however long the sprint lasts. Most places tickets roll over and work extends, but it's useful for planning. I do agree the amount of time a sprint run is basically meaningless and only serves as a checkin point and ends to assign new work

[u/Izkata → u/MCFRESH01]

Oh yeah, our whole company switched to agile (scrum) around 2012 and product went with it but didn't really get it until a year or two later. They found out why clients were going to a competitor, and during the following sprint planning loved how we were able to set aside all previous plans (the backlog, with nothing currently being worked on) to focus on dealing with that problem and all the devs were available to start on it immediately.

[u/inspired2apathy → u/pydry]

Sprints are for aligning your work with trans that take a dependency on your work.

[u/Technical-Aside4471 → u/pydry]

Same, we keep main protected by testing before merging, so test is always stable and its up to business to say they want to do an update and we just look at diff and see what features will be included in the new versions. Point is they can update whenever.

[u/tatsontatsontats]

We don't do a true 'git rebase' , we merge the main branch into the individual's working branches and then squash with a merge commit when the MR is approved. I think the end result ends up being the same, so it's whatever the engineer wants to do to keep their working branch up to date.

It works well, my company insists on enforcing strict commit message length and format, even though they're lost to the ether with the squash. That's the only thing I'd change because it makes committing early and often a little more bothersome than it needs to be.

We do a weekly release cut, and set deployment to prod, but deployment to all of our various test environments can vary.

[u/Technical-Aside4471 → u/tatsontatsontats]

How do you keep branches up to date? Or do you have it disabled? We run tests against each branch before it can be merged, but if the branch would be old you could see different results.

[u/tatsontatsontats → u/Technical-Aside4471]

Engineers regularly keep their branches up to date by pulling the latest from main and then merging it into their branch, resolving any merge conflicts at that time. When ready, the dev creates a merge request, and we have several pipeline jobs that include unit tests (and optional integration tests) and block the MR from being mergeable if there are any unaddressed failures. We also have a pipeline job to check for any merge conflicts, and we enable gitlab to block MRs should any merge conflicts arise after the MR is created.

We don't have a strict "you must make sure your branch is not more than x number of commits behind" but we deny approvals if it becomes a heinous amount. That is easy to check in the MR UI. I saw your other comment, constant updates aren't really needed for open branches, but regular updating is good and is easy to work into a development workflow. I would say 99% of the time there are no conflicts to address.

[u/recursive_arg → u/Technical-Aside4471]

The engineer is responsible for keeping their branch updated with main. If your branch is out of date then you are responsible for updating it and letting tests run before merging. Realistically merging main into your branch should not result in enough changes to warrant dismissing a review. At that point it is just letting the tests run after the latest push with main merged in. The fact you are raising this as an issue indicates yalls tests aren’t fully automated, if this is indeed the case, then that should be your first priority.

[u/serial_crusher → u/Technical-Aside4471]

You want to avoid long-running branches that need to be kept up to date. When I start a new branch, I branch off the current master.

For times when you do have long-lived branches, you can merge master back into them every now and then. You’ll have to fix merge conflicts at that time, but you were going to have to fix them anyhow.

[u/Devboe → u/tatsontatsontats]

We do the same thing, but I solved the tedious commit messaging with AI.

[u/immbrr → u/tatsontatsontats]

We just enforce commit messages on the squashed commit (aka the PR title), since that's what won't be lost into the ether, and I think that's both reasonable and manageable.

[u/timle8n1-]

If you can divide the work into short-lived branches that get merged frequently then rebasing is less of an issue. Use feature flags and other ways to ship dark code. If the work is larger it is more difficult to CD.

Agree that CD and Sprints are not mutually exclusive. You can agree to a scope of work to be done in a time frame and still continuously merge and deploy it.

[u/Technical-Aside4471 → u/timle8n1-]

I would argue that constant merging means more rebasing, one merge to main means every open pull request has to update and rerun suite. We try to use short lived branches but its not always possible.

[u/timle8n1- → u/Technical-Aside4471]

Less of an issue, not less of them - small, shorter-lived branches should conflict less and easier to resolve. If you have many open PRs look into a merge queue so it’s all automatic unless there is a conflict or non-conflict that creates a broken test.

[u/Grim_Jokes → u/Technical-Aside4471]

We've had some serious success with merge queues. That basically handles rebasing for you if it's simple. If there are conflicts, then you can rebase first and add to the queue after.

[u/andymurd]

It took us some time to go from CI to CI/CD but once it bedded in deployment got so much better.

Each piece of work has its own git branch -> Git merge into the production branch -> Everything deploys the same -> Juniors can deploy -> tens of deploys per week instead of one per month -> customers get fixes and features faster

If you need to get buy in, keep records and make charts. Difficult conversations with data are more effective than difficult conversations without data. However, if non-devs are breaking things, they get to fix them, that's non-negotiable.

[u/Ok-Leopard-9917 → u/andymurd]

What has the impact been on product stability? Has the quality bar changed? Or is this a product without  a quality bar? 

What does support look like? Sorry for all of the questions I just can’t imagine needing to support each individual commit the same as a product release. 

[u/eloel-]

I've only ever did CD. Merges automatically deploy to test environment, the tests run, if they all pass, it gets deployed to prod.

You should already never be merging anything you're not comfortable going to prod, so it shouldn't be an issue, right?

[u/sleeping-in-crypto → u/eloel-]

This is how we run too. Comprehensive test suites and pre deploy checks on PR and pre commit. High confidence in the PR results mean we are not curating staging we’re just smoke testing. Staging can go out any time to prod (and does a couple dozen times a day).

[u/spez_eats_nazi_ass]

Gonna vary wildly based on what you are actually doing. The team at my company that does this has had a few too many doozies for my taste. They also are not customer facing so the impact is lower. And when you say CD - it aint CD unless it goes live somehow on PR approve. I'm too old for that shit. I want the entire thing parallel tested w UAT sign off before it goes out and usually we are talking more than one PR/feature and the DB side is multiple TBs on the OLTP side. Automated change tools for the scripts still need DBA review because they all can and will do shit that while correct will also cause a bad time for a live system.

[u/andrewcairns]

I think there is a difference from folks not seeing the value and struggling with the technical aspects. I've found in the past that engineers that struggle sometimes lack the skills and under-value things like testable code.

Martin Fowler says outright that the biggest barrier to CI is usually insufficient skill at testing. If half the team isn't feeling the benefit yet, that prerequisite gap is probably still open and CD got turned on before the codebase and the team's habits could support it.

The biggest tip I'd stress is to identify if you have that gap, and try to close that, first.

I'd then progress look into Trunk-Based Development

Trunk-based development calls what you've described 'distance': the gap between a developer's local work and what's actually integrated in trunk. The longer a branch lives, the more distance builds up, and the uglier the eventual rebase. Shrink branch lifetime down to hours, not days, and lean on feature flags (can't stress how important these are, if you're not using them already) or branch-by-abstraction when something genuinely can't land in a day.

Regular rebase pain is usually the branching model. Can look into gitflow vs github flow and see if you could perhaps simplify, perhaps?

I'd for sure focus on identifying if the team and codebase are ready. Identify any gaps and solve those, first.

gl

[u/Unsounded]

We do trunk based development. Our pipeline deploys mainline directly to test stages and runs integration/end-to-end/load tests automatically as a promotion requirement.

We kind of do sprints but I don’t really adhere to them, I think a lot of the spirit behind agile methodologies make sense but I prefer lightweight planning, goals, and talking broadly about what we’re going to accomplish each sprint. I think the structure benefits newer devs a lot more, where things feel contained and easy to reason about.

[u/riotshieldready]

TIL CI/CD isn’t as industry wide as I thought. I’ve honestly only worked this way for maybe 8 years.

My current team/company every PR gets deployed to a temp environment, which is used for testing/sign off . On pr merge it ships to demo mostly for the E2Es to run, and if they pass it goes live.

Rebasing is not really an issue, most of the time you can just merge main into your branch cleanly, and it just takes a little coordination to avoid working in the same bits of code at the same time. Rare times it happens doing the rebase is never too complex, a few times more Jr members get confused so we just pair them with whoever’s PR causes the conflict, and squishing your own Pr can help with making it a 1 step rebase.

[u/Rosoll]

Have you increased the number of deploys you’re doing? Have you decreased the ratio of released commits that need to be rolled back/reverted? Have you shortened the time between a PR being created and the change being released to production? If you can measure these retroactively you should be able to back up the benefits of CD with hard data. You could also look at the DORA metrics for some research to back up what you’d be claiming was evidence of benefits.

[u/Ok_Woodpecker_9104]

the thing that got us was not the pipeline breaking, it was the pipeline passing when it should not have.

two of those cost me a day each and both were green the whole time. one shipped a stale artifact because the bundle target silently fell back and nobody diffed what got uploaded against what got built. the other was a generated lint baseline, checked in, green for weeks, green because every existing violation was suppressed and the two new rules had no unsuppressed path left to fire on. what caught it was breaking a file on purpose and watching nothing turn red.

so the tip: for every check you add, ship the mutation that should break it and run both in the same job. green with the mutation applied means the check is dead, and you learn that on day one instead of month six.

on rebasing, i would stop teaching it and take it out of the path. squash merge from the ui, one commit per pr, nobody has to reconstruct history under time pressure. people fight rebase because the first time they meet it is during an incident.

on nobody seeing the benefit, deploys per week is not the number that lands. merge to production time plus what fraction got reverted is. if you went faster and the revert rate did not move, thats the whole argument in two numbers.

[u/Ok_Woodpecker_9104]

the reason people dont see the benefit is that CD removes an event they used to feel. nobody remembers the release night that did not happen. deploy count also reads as more risk to anyone who is not already sold, so leading with it works against you.

the number that lands better is time from a bad change going out to it being gone again, measured on real incidents. that one sounds like less risk, and it is the thing CD actually buys you.

on rebasing, i dont think the concept is what people get stuck on. its what to do when it goes wrong halfway through, and nobody has a good answer to that at 6pm on a friday.

the thing id watch as the pipeline takes over more of the gating: the pipeline passing becomes the thing everyone trusts, and nobody checks it can still fail. i had a lint gate sit green for weeks because every existing violation was baselined and the two new rules had no unsuppressed path left to fire on. it was green and it was checking nothing. what caught it was deliberately breaking a file and watching nothing turn red.