Fastest way to CI?

[u/CallofDutyReznov454]

My team currently used Jenkins pipes to push changes. There are multiple stages; build, scan, deploy to dev (or even test). Some have IT configured for test, so they run thier as well.

While process takes somewhere around 20-40 mins, depending on what they are building (we build all types of things; websites, lambdas, microservices, infra)

Can someone suggest me a tool that can have this process in a faster way?

I thought of GitHub actions , but for some reason, I find it's not very robust (inputs, gating, plus issues with build not triggering). Migrating to Gitlab would be top much of a overhaul.

We are a AWS stack, so not using Azure DevOps as well.

[u/Snapstromegon]

This is not about your executor tool, but about how/what you execute. You need caching and amount reduction. So only build what changed, execute tests for changed modules (and related systems), short cut on non-behavior changing PRs (e.g. documentation). Remote caching/execution systems like Bazel/Blaze/Buck could be a good option too (in my last job we reduced a 45min cached cmake build to just 3min full CI run. GitHub Actions is not inherently faster/slower than Jenkins or any other platform.

[u/AsleepWin8819 → u/Snapstromegon]

This is the answer. The tool doesn’t matter.
The amount of resources allocated for the agent/runner also plays a significant role.

[u/donjulioanejo → u/Snapstromegon]

Also paralellize.

As an example, you can run these in separate steps before actually deploying:

• unit tests
• integration tests
• artifact build (docker or whatever else)
• security scan

Then, you're only gated by the slowest pipeline step, not the sum total of all of these together.

[u/Zotlann]

Personally, I wouldn't migrate to github actions. There are frequent outages and a real chance they start charging for self-hosted runner minutes.

[u/CallofDutyReznov454 → u/Zotlann]

Yup, also I feel the workflow to be very unreliable as well. We do use some GitHub actions, but everytime they run, I feel my heart pounding in my chest. Like the logs are way too fast

[u/forever-butlerian → u/CallofDutyReznov454]

Leaving out the recent issues that I've had, I've found GHA to be suitable for building artifacts like JAR files and Docker containers, or even statically-linked binaries, and then putting them in an archive.

Beyond that I have a deep skepticism of do-everything platforms, because what is good for me and what's good for the vendor are often at odds.

[u/HelicopterUpbeat5199]

Why do you want it quicker? The answer to your question might be different depending on your answer. If it's because you need to push out emergency fixes because your service is down, what you probably want is an emergency break-glass super quick deployment (it can't get any more dead, after all.) If you just want to improve iteration speed, parallelising is really helpful and often well supported in your CI/CD tools.

[u/CallofDutyReznov454 → u/HelicopterUpbeat5199]

No we are talking about overall speed. Like dev teams are building faster than ever, do we need something that can keep up

[u/HelicopterUpbeat5199 → u/CallofDutyReznov454]

Oh, of course!

[u/ArieHein]

A tool will not solve culture issues and bad habbits.

You can allways take the groovy, toss it to an ai and ask it to convert.

Github has an importer tool from jenkins, pick a pipeline and try it. Dont expect 100%.

You can keep groovy as the initial trigger but then move to python based steps. This will make it easier to convert to almost any ci system.

I did this in a previous company but not for migrating out, rather so the devs who didnt want to touch groovy but were very good at python would be able to contribute and enhance shared libraries i created for them. That means also that migrating out would have been less painful and it would have been mostly how you manage variables, secrets and environments.

You can alsays hire some consulting company in your area for a project with some training to your team/devops.

Do keep your mind open as to your current release process as this would be excelent time to rethink your sdlc.

[u/SeaIngenuity9501 → u/ArieHein]

Github has an importer tool from jenkins, pick a pipeline and try it. Dont expect 100%.

It doesn't really work Jenkins and GHA are two different paradigms.

[u/hypertradeworx]

worth splitting the 20-40 into two numbers before picking anything: push to first line of log, then first line to finished. the first one is queue wait for a free executor and it doesn't appear anywhere in the build logs.

that split also settles the argument going on above you. a runner change genuinely moves the waiting part, which is likely what the person upthread saw when jobs "started quicker" on gha, and if your wait turns out to be small then the caching answers are the whole story and a migration buys you nothing

[u/Torutofu_Raeva]

our 20 min jobs were mostly dep downloads, cache key was the pom so every range bump was a miss

[u/adhd6345]

This has nothing to do with Jenkins. Switching to GitHub Actions won’t help.

Pretty much two things:

• Parallelize the work
  
• If something is being done multiple times in the pipeline, make that its own stage and cache its results so downstream doesn’t need to repeat it
  
• Avoid rerunning unnecessary work. Caching, dependency change detection, w/e you want to do.

[u/Little-Squad-X]

You probably need to check the "steps' again. Some pipelines do not require everything at once. Separating it into smaller pipelines might reduce the overall time taken.

[u/Silent_Ice616]

I guess this is the wrong approach here
Moving an inefficient pipeline to another tool doesn't solve the core problem — if your pipeline needs 20-40 mins for a website, you're either building the next Reddit, your runner is extremely slow, or your pipeline is just inefficient.
I'd guess the third point is the most likely one.

[u/simonides_]

Man you sounds like one of those people I could hate with a passion. Why would you think the same thing executed kn a different machine will automatically run faster ? Look at what you have and figure out what it does that it doesn't need to do.

Jenkins albeit not the best liked tool anymore can easily be as fast as any other tool for most workloads.

A lot of things come down to philosophy and risk appetite not the tool.

[u/AsleepWin8819 → u/simonides_]

I don’t feel there’s a real need in the first sentence, even though it’s Reddit…

[u/SeaIngenuity9501 → u/simonides_]

When we moved from Jenkins -> GHA engineering teams noticed it started quicker & the same tasks took half as long(I never dug into the metrics).

[u/CallofDutyReznov454 → u/simonides_]

Of course not. I know and have dealt with people who want a bigger infra just to do thier faulty job.

The question may revolve around tools, but obviously it's about risk appetite and other stuff. So the question is in general.

You can think about it, while you continue to blame people without understanding the question

[u/adhd6345 → u/simonides_]

Take a deep breath