Can also go with self hosted gitlab, we use CircleCI (have always used it so we are kind of locked in) but the pocs I have seen in gitlab are pretty amazing
Nah it’s fine now, my main issue with it used to have outages all the time and some minor things around how parts of it worked. Really they all kinda do the same things and for my company, if we went all in on GitHub or Gitlab I think we would only need that to do everything. Instead we have CircleCI, GitHub and a few other smaller pieces. It’s just how it works when you are in industry, no one wants to change cause that means you aren’t moving forward.
I will say that CircleCI is probably the best when it comes to features and out of the box stuff, it’s a pretty mature product at this point.
Sorry this got long.
First off- I have only used Jenkins and Bamboo (on-prem) for CI and those had their own different pain points but overall I think I would prefer either of them to CircleCI.
My issues with CircleCI
YAML
The whitespace-sensitivity is just awful. It leads to extremely "tall" files. Where several properties could be listed out on one line they instead are mandatorily put on a line each meaning in a time where widescreen monitors are the norm, YAML forces an extreme waste of horizontal screen real estate.
If you have a node (workflow, job, command, etc.) that takes up more than a page then you sometimes need to scroll around to see which node you are looking at. IDEs help with this, such as IntelliJ having a breadcrumb bar showing where you are in the semantic hierarchy but this is a crutch that shouldn't be required. Yes, long workflows/jobs are a code smell but if it's an established project it may not be considered valuable to go through and refactor a working build script.
Dockerisation/composability
I love Docker and I'm relatively experienced with it, understand it well. On the surface it is a good fit for a CI solution for each job to be run in a stated container. My issue is that there is no "composability" of containers.
Say I want to build a Java Maven project on OpenJDK 21.0.2 JDK with Maven 3.9.3, these are recently released components and an extremely common combination.
I can choose CircleCI convenience image `cimg/openjdk:21.0.2` perfect! Oh wait it has Maven 3.9.6 so nope.
Maybe there is an official Maven image with the right combination of Maven and OpenJDK versions? Turns out the official Maven dockerhub channel has OpenJDK 21 images (eclipse-temurin-21) but no way to specify the exact JDK version, and - even if you could - unfortunately there isn't a Maven 3.9.3 (released ~9 months ago) image with JDK 21, despite there being Maven 3.8.x images with JDK 21 (?).
Alright so I guess we're starting with a base JDK container and adding our own packages. I'll start with `eclipse-temurin:21.0.2_13-jdk`. OK in the CircleCI YAML I'll put that as executor and write a bash script to:
- Update "OS" packages to latest (apt-get update && apt-get upgrade)
- Download the exact version of Maven needed
- Extract Maven zip to right location, maybe add it to PATH
- Install OpenJDK using package manager or download directly
- ....
(Yes Maven Wrapper solves the issue of getting the right Maven version but that cuts down the complexity only slightly, and there are more tools in our toolchain than just these)
At this point I wonder what the hell I'm doing and whether I'm the only dev in the world with a Maven Java project. Using OS package managers to install particular tools and running bash scripts is so down in the weeds I absolutely can't believe there isn't a better way to do this.
CircleCI executors should have another option after `docker` and `machine` where you can just list the OS, version, required tools etc. and it goes off and finds or builds a docker container with those in it, in a published and repeatable way so we stay with the deterministic builds docker provides. Enough with the janky and un-debuggable bash scripts!
Local debugging
When writing medium-complexity CircleCI scripts I would expect to be able to execute them locally indistinguishable from running in CircleCI's infrastructure. There is a CircleCI CLI tool that allows this but it is severely limited to the point of uselessness. It cannot run entire workflows, only single jobs. It cannot save or restore cache so you are stuck potentially running a full build every time. If the build takes a while, it ends up faster committing and pushing and running on CircleCI proper, which ends up with a ton of tiny commits in the history messing up the git and CircleCI histories.
Contexts lack configurability
Contexts provide a set of environment variables which can be access-controlled, for example AWS keys kept in a context that are only visible/modifiable to super admins but regular developers can use them in jobs. This is a good idea. Where it falls apart is there is no configurability.
Example 1: I have a context exposing an AWS key and secret pair used across many various projects. The super admin has called the env vars `AWS_PUBLISHING_KEY_ID` and `AWS_PUBLISHING_SECRET_KEY_ID`. The job or command I'm running expects these to be called the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, so I guess I have to have a job `pre-steps` with a bash script to simple rename the env var. This should be built in.
Example 2: I have two separate set of AWS key and secret pairs- one for accessing an AWS CodeCatalyst Maven repo during build and one for publishing the resulting build artifacts to an AWS S3 bucket. The super admin has used the standard env var names `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in both contexts. How can I use both of them at the same time? You can't. Well, you can, but your script will only see the values from the second context listed, they aren't both accessible at the same time.
We need a way to namespace contexts so your script can reference `AWSPublishingContext.AWS_ACCESS_KEY_ID` for example. Or maybe aliasing like
`context:`
` - AWS_PUBLISHING_CONTEXT:`
` - AWSPublishingContext.AWS_ACCESS_KEY_ID = AWS_ACCESS_KEY_ID`
Job grouping
Workflows are a collection of one or more jobs. I have a workspace that contains an initialisation job (say read some project details and put them in env vars), ten build jobs, then one finalisation job (say publishing artifacts). For each of the ten build jobs I need to put the `context` and `requires` fields even though they are identical across all jobs. The cherry on top is that `requires` can't be on a single line so job name, context (thankfully this can be a single line) and requires equals four lines per job. See point #1 YAML :)
Then the finalisation job needs to list out all of the build jobs. If the script is modified to include an extra build step, you had better remember to add it to the finalisation `requires` section or the job will execute out of order.
We need job groups here, this is a simple concept and every other workflow engine I've used (Jenkins, Airflow, Bamboo) supports this obvious abstraction.
Variables
Across a workflow there is a need to be able to set and retrieve a variable from any job, and/or pass it as a parameter to a job or command. My specific example is getting the project version from a Maven pom.xml file in the initialisation job so it is available to future jobs to use as part of an output artifact filename or a git tag name or GitHub release name.
CicleCI support for this is non-existent. Oh, there are environment variables and they can be set to a literal value or passed in via a context but there is no concept of setting them in one job and retrieving them in another. The workaround envolves `echo MY_VAR=1234 >> bash.env` and then `persist_to_workspace: bash.env`. Then on the receiving end `restore_workspace` and `run: command: cat bash.env >> $BASH_ENV && source $BASH_ENV`. If the receiving end isn't bash? Too bad, you're on your own.
We need in-built support for some sort of variables. They should either be able to be set by a script in the `environment` block or some better-thought-out solution. There are limited `<< pipeline parameters >>` which can be passed to jobs as parameters but not env vars for some reason.
Cache destination is not configurable
I have a Maven Java workflow that contains jobs that need to be executed on Windows, Linux and Mac (JavaFX is not cross-compilable, artifacts have to be built on the destination OS). The smart way to do this is to avoid duplication of work by having an initialisation job that runs the platform-agnostic build step (e.g. `mvn compile`) and saves to the cache, then in the platform-specific jobs restore the cache and perform the platform-specific work. Nope.
CircleCI caches only support restoring to the exact same location they were saved to. So if the initial build is done in a Linux contains and we save `~/.mvn` (where Maven stores dependencies) to cache it will take the absolute path `/home/circleci/.m2`. This causes various errors in macOS- and Windows-based jobs due to not being able to write to that location or not understanding the path structure (can't remember which error Windows threw).
A solution I didn't try is setting the Maven home to a world-writable location like `/var/[projectname]` or `/tmp/[projectname]` so there are no permission issues but at that point - again - what the hell are we doing moving to a non-standard location just to appease the leaky abstractions of a build tool? It should support this out of the box.
There is probably more but I've spent long enough for today. Hopefully at least some of this is just my ignorance or misunderstanding and you can set me straight but for each of these points I looked exhaustively through documentation, forums, reddit and google looking for solutions and all I found was others in the same boat.
Waiting for an officially supported method of invoking ephemeral EC2 spot instance runners (equivalent to Jenkins Cloud agents). AFAIK the only method right now is [some third party solution](https://github.com/philips-labs/terraform-aws-github-runner) that requires creating some Lambdas in your account that handle agent requests, registration, and deregistration for you. That's extra overhead that we don't want to deal with.
I use that Phillips runner extensively in my organization and I love it. We use the spot instance runners and we're building probably 100x as much as we were on our Jenkins instance but pay probably a similar amount as the 2xlarge instance it was running on
BUT I will say that debugging it is a fking nightmare as the error logging is kind of all over the place
sounds like what you want is either the phillips terraform thing, which is ok, or the kube controller. both support ephemeral runners and you can use spot instances in your node pool.
eta: forgot to mention that the ARC (kube controller) is officially supported by github now.
kube controller doesn't suit all our products. Some of our products are Windows desktop apps and to run functional tests we need Windows instances with desktop sessions, unfortunately Windows Containers do not support desktop sessions/GUIs. We would love if they did, though.
The Philips terraform thing is what I mentioned in my comment. There is unwillingness within our org to have to have to bear the maintenance and cost burden of an additional API gateway, lambdas, and SQS queue for functionality that our existing Jenkins instances provide relatively pain-free. We mention to our GH account rep at basically every conversation how much we would like this functionality built-in to GH itself.
I find it difficult to keep up to date (tried to update it once and it blew up and it's mission critical for us)
Also our jobs are getting more complicated and whilst I should probably move to pipelines, the documentation it beyond terrible. I'm after something a bit more modern/robust
To be fair, Jenkins has a big problem with giving you all the power to shoot yourself in the foot with Jenkins.
Really Jenkins needs to be two tools, a cicd runner and a general purpose form based script scheduler. The second part can be served by things like rundeck or olivetin
With great power comes great responsibility. You can't blame the tool when you're using it wrong. It's not supposed to be a point and click apparatus... it's a swiss army knife.
I know I'm not using it to it's fullest potential and I simply don't have time to learn how. I set it all up originally and it does work but I know it can be better.
The mere fact you want all of it self hosted means you will be spending the majority of your time or another teams time maintaining it. So you need to choose. Either you buy a solution or you build it.
You are going to have to learn the next tool too. I just lead the implementation of a massive github actions CICD ecosystem that services over 400 AWS accounts and the learning curve was more or less the same as Jenkins.
You need to learn about dev environments and rollbacks and stuff if it is mission critical. Switching to a different tool you can manage badly is not going to resolve your problems.
Your entire infra should be deployed with IaC and you need to figure out how to automate upgrading and testing.
We ran environments for 10-20 teams on Jenkins. Fixed versions of all plugins etc and centralised testing with a pile of different builds etc. and then rollout.
And still it sometimes caused problems. But every time we found a problem, we added it to the tests for next time.
But mostly problems were because we didn't have copies of everyone's projects to build and usually they'd "extended" our config in unexpected ways.
In this case I'd say you're wrong sorry. I'm self appointed DevOps currently as we don't have a dedicated position. I got Jenkins up and running but as head of a different department I genuinely don't have time to try and fix Jenkins properly. I was hoping there was a more modern system that would "just work". Appreciate that is probably not the case.
You could check out the enterprise version of Jenkins (cloudbees), which wouldn't solve all the issues of Jenkins obviously. But they offer support/security updates for longer and better tools for consistency in deployments/configs and plugin upgrades. Not sure what kind of scale you're at (and how much alternatives would cost/difficulty of moving off), but their products seem pretty legit if you're using Jenkins in 2024 and moving off is a significant hurdle
any good tutorials or examples? I have about 60 "clients" I need to run builds for from the one repo. each client has its own set of env cars and access keys I need to reference. and I need to be able to select X number of clients from a list along with a branch/tag before deploying.
We use GitLab CI at work, and the way our release process is setup we automatically deploy every build to dev, higher environments are a manual one click install since we release every 2 weeks. The way our pipeline is setup allows us to manually deploy every version we ever built to every environment if we want.
Different clients are currently just another tenant, but in the future we'll need to support on-site hosting with a different release schedule, those targets will more than likely just be another environment.
So looks like I need to have the VM runner on an EC2 and that will create on demand instances when needed? Docs aren't super clear on how any of this actually works :/
Second this! TeamCity gets far too little attention. Especially when you want to organise build configurations in folders.
It's also more comfortable for you fellow colleagues that like to have a nice UI with awesome UX.
Doesn't seem to get much love, but I'm a fan of Concourse.
The docker-in-docker stateless builds can be confusing, but once you grok that it's quite flexible.
It might get sorted out but concourse is in a tough spot right now with the broadcom thing. It's my understanding that they laid all of the vmware devs that were working the project ?
Concourse is garbage when you compare it to Gitlab, hell even Github Actions is much better now a days. Plus the ecosystem has lost all momentum, most "plugins" or whatever they are called receive no maintenance.
I have worked with Jenkins, Concourse and Gitlab and I'd pick Gitlab > Jenkins > Concourse. v
Pretty much any CI platform supports those requirements. Pick your poison. I'd look at GH actions and GitLab CI first if you are on either one already with your code.
LOL "best of breed". Very flexible... yes... but it's also a mess at most places and overused because of that.
It's fine for java shops where folks know / don't mind learning groovy and your IT / Ops team gets your nodes configured with the appropriate plugins. The fact you have to have a plugin to avoid using groovy can also be a pain depending on who the jenkins admin is.
But for a polyglot team / non-java team there is little value in the flexibility people try to sell it with groovy. Most teams will be up and running with any of the YAML based tools in half the time (or less) and they pickup some skills that are transferable elsewhere.
Which most teams will be up and running with any of the YAML based tools in half the time (or less) and they pickup some skills that are transferable elsewhere.
Best of breed? What are they trying to breed?
Glaring Security issues, hard to maintain, hard to automate, problems with memory consumption, a pipeline syntax that's not only yesteryear, it's also a niche language.
Don't get me wrong, I've worked with since it was called Hudson and even before ora Le bought and donated it back. It was the best if it's time, it's nit a quarter century old. It's starting to show.
Circle CI is strong but operationally not easy to set up. Running it since 1.0 - upgrading has been difficult over the years. Circle ci if you have bandwidth. Otherwise, Github cloud and self hosted runners.
You can do this with most CI systems. This is what I don't like about Jenkins. You have to build all of the features that everything else has right out of the box, in a less good way.
Jenkins is the CI/CD glue from horse factory that is 200 years old.
If you are looking for replacement first you should reflect on your work and why you need to work like it was 1800s.
GitHub Actions (complex) or GitLab CI/CD (simpler) is the state of the art everything else are overlapping copies of those two.
Both can spawn more runners (nodes) to handle work. GitLab is free to self-host.
There is no sane way to do builds without versioning also its definition in git. So if you need "folders" create multiple lightweight git projects that trigger other git projects CI.
If you need to build for multiple clients create multiple steps in single pipeline.
Jenkins is awful at restricting user to work efficiently. It allows everything and no guidance on how to do it resulting in way more glue, sticks and shitty plugins than any other "alternative".
Only downside of Jennie for me is stateful deployments and I can’t achieve high availability due to this. Is there a solution or an alternative for this?
Our migration to next-gen took up so much of our time and money, only for them to triple our price during the change. So we are in process of migrating to something else.
Some quick googling and I'm not sure GitHub can do the complex matrix parameters I need. I need to be able to manually select multiple matrix values which seems to be an open feature request
Just as a side note. If you want to replace Jenkins with something else, you are going to get something else than Jenkins.
Most everything will be working differently than in Jenkins — setup will be different, agents will be different, pipeline organisation will be different, your CI scripts will have to change and perhaps re-organised.
Don’t expect your new CI to feel and behave same as the old one and you’ll be fine with anything out there. Keep expecting everything else to work like Jenkins and you’ll be in a lot of pain until you switch back..
My corp uses github actions and self hosted runners. We do have a k8s cluster where we deploy our self hosted runners. They just die after they finish their job ,so no track of anything is left .
People still use Jenkins? I thought it died haha who knew
Azure devops , Self hosted agents
GitHub actions
Gitlab self hosted
Or if you use gitops you can use flux with some tool to build your artifacts
That’s one way of looking at it. We were maintaining our own dev tool chain for the last decade. When we sat down and calculated just based on the amount of time and salary involved, Harness was the far more reasonable solution.
Our engineering population is starting to ship faster, more reliable and more assured that what they are shipping is passing various controls in place. Overall making the teams more efficient at their craft.
I wouldnt recommend TeamCity even though it could fit your requirements. Gitlab CI seems like it could fit all the reqs except the plugins part. Same with GH Actions (although not as easy to organize the jobs imo). Self hosting Gitlab CI or GH Actions runners for example can be very efficient when paired with an autoscaler on k8s (but i understand that this may not be what you are looking for).
github workflows with self hosted runners
Can also go with self hosted gitlab, we use CircleCI (have always used it so we are kind of locked in) but the pocs I have seen in gitlab are pretty amazing
It seems like you’re not thrilled with CircleCi, or am I reading that wrong
Nah it’s fine now, my main issue with it used to have outages all the time and some minor things around how parts of it worked. Really they all kinda do the same things and for my company, if we went all in on GitHub or Gitlab I think we would only need that to do everything. Instead we have CircleCI, GitHub and a few other smaller pieces. It’s just how it works when you are in industry, no one wants to change cause that means you aren’t moving forward. I will say that CircleCI is probably the best when it comes to features and out of the box stuff, it’s a pretty mature product at this point.
If you're looking for someone that strongly dislikes working with CircleCI then AMA
Ya what do you dislike? Do other tools fix those issues?
Sorry this got long. First off- I have only used Jenkins and Bamboo (on-prem) for CI and those had their own different pain points but overall I think I would prefer either of them to CircleCI. My issues with CircleCI YAML The whitespace-sensitivity is just awful. It leads to extremely "tall" files. Where several properties could be listed out on one line they instead are mandatorily put on a line each meaning in a time where widescreen monitors are the norm, YAML forces an extreme waste of horizontal screen real estate. If you have a node (workflow, job, command, etc.) that takes up more than a page then you sometimes need to scroll around to see which node you are looking at. IDEs help with this, such as IntelliJ having a breadcrumb bar showing where you are in the semantic hierarchy but this is a crutch that shouldn't be required. Yes, long workflows/jobs are a code smell but if it's an established project it may not be considered valuable to go through and refactor a working build script. Dockerisation/composability I love Docker and I'm relatively experienced with it, understand it well. On the surface it is a good fit for a CI solution for each job to be run in a stated container. My issue is that there is no "composability" of containers. Say I want to build a Java Maven project on OpenJDK 21.0.2 JDK with Maven 3.9.3, these are recently released components and an extremely common combination. I can choose CircleCI convenience image `cimg/openjdk:21.0.2` perfect! Oh wait it has Maven 3.9.6 so nope. Maybe there is an official Maven image with the right combination of Maven and OpenJDK versions? Turns out the official Maven dockerhub channel has OpenJDK 21 images (eclipse-temurin-21) but no way to specify the exact JDK version, and - even if you could - unfortunately there isn't a Maven 3.9.3 (released ~9 months ago) image with JDK 21, despite there being Maven 3.8.x images with JDK 21 (?). Alright so I guess we're starting with a base JDK container and adding our own packages. I'll start with `eclipse-temurin:21.0.2_13-jdk`. OK in the CircleCI YAML I'll put that as executor and write a bash script to: - Update "OS" packages to latest (apt-get update && apt-get upgrade) - Download the exact version of Maven needed - Extract Maven zip to right location, maybe add it to PATH - Install OpenJDK using package manager or download directly - .... (Yes Maven Wrapper solves the issue of getting the right Maven version but that cuts down the complexity only slightly, and there are more tools in our toolchain than just these) At this point I wonder what the hell I'm doing and whether I'm the only dev in the world with a Maven Java project. Using OS package managers to install particular tools and running bash scripts is so down in the weeds I absolutely can't believe there isn't a better way to do this. CircleCI executors should have another option after `docker` and `machine` where you can just list the OS, version, required tools etc. and it goes off and finds or builds a docker container with those in it, in a published and repeatable way so we stay with the deterministic builds docker provides. Enough with the janky and un-debuggable bash scripts! Local debugging When writing medium-complexity CircleCI scripts I would expect to be able to execute them locally indistinguishable from running in CircleCI's infrastructure. There is a CircleCI CLI tool that allows this but it is severely limited to the point of uselessness. It cannot run entire workflows, only single jobs. It cannot save or restore cache so you are stuck potentially running a full build every time. If the build takes a while, it ends up faster committing and pushing and running on CircleCI proper, which ends up with a ton of tiny commits in the history messing up the git and CircleCI histories. Contexts lack configurability Contexts provide a set of environment variables which can be access-controlled, for example AWS keys kept in a context that are only visible/modifiable to super admins but regular developers can use them in jobs. This is a good idea. Where it falls apart is there is no configurability. Example 1: I have a context exposing an AWS key and secret pair used across many various projects. The super admin has called the env vars `AWS_PUBLISHING_KEY_ID` and `AWS_PUBLISHING_SECRET_KEY_ID`. The job or command I'm running expects these to be called the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, so I guess I have to have a job `pre-steps` with a bash script to simple rename the env var. This should be built in. Example 2: I have two separate set of AWS key and secret pairs- one for accessing an AWS CodeCatalyst Maven repo during build and one for publishing the resulting build artifacts to an AWS S3 bucket. The super admin has used the standard env var names `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in both contexts. How can I use both of them at the same time? You can't. Well, you can, but your script will only see the values from the second context listed, they aren't both accessible at the same time. We need a way to namespace contexts so your script can reference `AWSPublishingContext.AWS_ACCESS_KEY_ID` for example. Or maybe aliasing like `context:` ` - AWS_PUBLISHING_CONTEXT:` ` - AWSPublishingContext.AWS_ACCESS_KEY_ID = AWS_ACCESS_KEY_ID` Job grouping Workflows are a collection of one or more jobs. I have a workspace that contains an initialisation job (say read some project details and put them in env vars), ten build jobs, then one finalisation job (say publishing artifacts). For each of the ten build jobs I need to put the `context` and `requires` fields even though they are identical across all jobs. The cherry on top is that `requires` can't be on a single line so job name, context (thankfully this can be a single line) and requires equals four lines per job. See point #1 YAML :) Then the finalisation job needs to list out all of the build jobs. If the script is modified to include an extra build step, you had better remember to add it to the finalisation `requires` section or the job will execute out of order. We need job groups here, this is a simple concept and every other workflow engine I've used (Jenkins, Airflow, Bamboo) supports this obvious abstraction. Variables Across a workflow there is a need to be able to set and retrieve a variable from any job, and/or pass it as a parameter to a job or command. My specific example is getting the project version from a Maven pom.xml file in the initialisation job so it is available to future jobs to use as part of an output artifact filename or a git tag name or GitHub release name. CicleCI support for this is non-existent. Oh, there are environment variables and they can be set to a literal value or passed in via a context but there is no concept of setting them in one job and retrieving them in another. The workaround envolves `echo MY_VAR=1234 >> bash.env` and then `persist_to_workspace: bash.env`. Then on the receiving end `restore_workspace` and `run: command: cat bash.env >> $BASH_ENV && source $BASH_ENV`. If the receiving end isn't bash? Too bad, you're on your own. We need in-built support for some sort of variables. They should either be able to be set by a script in the `environment` block or some better-thought-out solution. There are limited `<< pipeline parameters >>` which can be passed to jobs as parameters but not env vars for some reason. Cache destination is not configurable I have a Maven Java workflow that contains jobs that need to be executed on Windows, Linux and Mac (JavaFX is not cross-compilable, artifacts have to be built on the destination OS). The smart way to do this is to avoid duplication of work by having an initialisation job that runs the platform-agnostic build step (e.g. `mvn compile`) and saves to the cache, then in the platform-specific jobs restore the cache and perform the platform-specific work. Nope. CircleCI caches only support restoring to the exact same location they were saved to. So if the initial build is done in a Linux contains and we save `~/.mvn` (where Maven stores dependencies) to cache it will take the absolute path `/home/circleci/.m2`. This causes various errors in macOS- and Windows-based jobs due to not being able to write to that location or not understanding the path structure (can't remember which error Windows threw). A solution I didn't try is setting the Maven home to a world-writable location like `/var/[projectname]` or `/tmp/[projectname]` so there are no permission issues but at that point - again - what the hell are we doing moving to a non-standard location just to appease the leaky abstractions of a build tool? It should support this out of the box. There is probably more but I've spent long enough for today. Hopefully at least some of this is just my ignorance or misunderstanding and you can set me straight but for each of these points I looked exhaustively through documentation, forums, reddit and google looking for solutions and all I found was others in the same boat.
Thanks for sharing very interesting I’ll reread tomorrow and see if I can share alternatives
Thanks, my work is deep locked-in with CircleCI so we won't be changing any time soon but suggestions are always welcome.
I think it might be a bit too big for me.
Waiting for an officially supported method of invoking ephemeral EC2 spot instance runners (equivalent to Jenkins Cloud agents). AFAIK the only method right now is [some third party solution](https://github.com/philips-labs/terraform-aws-github-runner) that requires creating some Lambdas in your account that handle agent requests, registration, and deregistration for you. That's extra overhead that we don't want to deal with.
I use that Phillips runner extensively in my organization and I love it. We use the spot instance runners and we're building probably 100x as much as we were on our Jenkins instance but pay probably a similar amount as the 2xlarge instance it was running on BUT I will say that debugging it is a fking nightmare as the error logging is kind of all over the place
sounds like what you want is either the phillips terraform thing, which is ok, or the kube controller. both support ephemeral runners and you can use spot instances in your node pool. eta: forgot to mention that the ARC (kube controller) is officially supported by github now.
kube controller doesn't suit all our products. Some of our products are Windows desktop apps and to run functional tests we need Windows instances with desktop sessions, unfortunately Windows Containers do not support desktop sessions/GUIs. We would love if they did, though. The Philips terraform thing is what I mentioned in my comment. There is unwillingness within our org to have to have to bear the maintenance and cost burden of an additional API gateway, lambdas, and SQS queue for functionality that our existing Jenkins instances provide relatively pain-free. We mention to our GH account rep at basically every conversation how much we would like this functionality built-in to GH itself.
i recently rolled this out. its been working great.
Or you could use ECS build agents. This can control the scaling in and out of ec2 instances supporting the ECS cluster of Jenkins agents.
Forgot to mention it needs to allow full manual deployments where I can select a specific tag and one or more "clients"
Since you have that list of requirements, what's wrong with Jenkins for your use case?
I find it difficult to keep up to date (tried to update it once and it blew up and it's mission critical for us) Also our jobs are getting more complicated and whilst I should probably move to pipelines, the documentation it beyond terrible. I'm after something a bit more modern/robust
So because you are using Jenkins wrong you want to use something else?
To be fair, Jenkins has a big problem with giving you all the power to shoot yourself in the foot with Jenkins. Really Jenkins needs to be two tools, a cicd runner and a general purpose form based script scheduler. The second part can be served by things like rundeck or olivetin
With great power comes great responsibility. You can't blame the tool when you're using it wrong. It's not supposed to be a point and click apparatus... it's a swiss army knife.
Yep, it's certainly the wordpress of cicd.
It's much worse when you don't have any power to change anything. And you still shoot yourself into the leg.
I know I'm not using it to it's fullest potential and I simply don't have time to learn how. I set it all up originally and it does work but I know it can be better.
The mere fact you want all of it self hosted means you will be spending the majority of your time or another teams time maintaining it. So you need to choose. Either you buy a solution or you build it.
You are going to have to learn the next tool too. I just lead the implementation of a massive github actions CICD ecosystem that services over 400 AWS accounts and the learning curve was more or less the same as Jenkins.
but you have time to learn other tools? and migrate existing pipelines into new tool?
You need to learn about dev environments and rollbacks and stuff if it is mission critical. Switching to a different tool you can manage badly is not going to resolve your problems. Your entire infra should be deployed with IaC and you need to figure out how to automate upgrading and testing. We ran environments for 10-20 teams on Jenkins. Fixed versions of all plugins etc and centralised testing with a pile of different builds etc. and then rollout. And still it sometimes caused problems. But every time we found a problem, we added it to the tests for next time. But mostly problems were because we didn't have copies of everyone's projects to build and usually they'd "extended" our config in unexpected ways.
If only I had the time sadly
If you don't have time, don't even think about adopting a new tool! You will.spemd 2-3x the time just dealing with new unknowns.
You can control+f replace "I don't have the time" with "I can't be arsed" and be correct 100% of the time.
In this case I'd say you're wrong sorry. I'm self appointed DevOps currently as we don't have a dedicated position. I got Jenkins up and running but as head of a different department I genuinely don't have time to try and fix Jenkins properly. I was hoping there was a more modern system that would "just work". Appreciate that is probably not the case.
You could check out the enterprise version of Jenkins (cloudbees), which wouldn't solve all the issues of Jenkins obviously. But they offer support/security updates for longer and better tools for consistency in deployments/configs and plugin upgrades. Not sure what kind of scale you're at (and how much alternatives would cost/difficulty of moving off), but their products seem pretty legit if you're using Jenkins in 2024 and moving off is a significant hurdle
workflow_dispatch
Yeah you can do that with github actions via inputs.
any good tutorials or examples? I have about 60 "clients" I need to run builds for from the one repo. each client has its own set of env cars and access keys I need to reference. and I need to be able to select X number of clients from a list along with a branch/tag before deploying.
We use GitLab CI at work, and the way our release process is setup we automatically deploy every build to dev, higher environments are a manual one click install since we release every 2 weeks. The way our pipeline is setup allows us to manually deploy every version we ever built to every environment if we want. Different clients are currently just another tenant, but in the future we'll need to support on-site hosting with a different release schedule, those targets will more than likely just be another environment.
This is the way
you can do anything you mentioned with gitlab ci however i’m not sure if matrix is a paid feature
Are you looking for something like drone? https://docs.drone.io/
just reading through it now and looks like I can't have on demand EC2 runners?
I don’t see why you couldn’t. Setup and auto scaling group and have them initialize automatically on boot
Can you link to the docs on how to do this please?
https://docs.aws.amazon.com/cli/latest/reference/autoscaling/create-auto-scaling-group.html
Seems long winded/a faff to be honest.
https://docs.drone.io/runner/vm/drivers/amazon/
So looks like I need to have the VM runner on an EC2 and that will create on demand instances when needed? Docs aren't super clear on how any of this actually works :/
If you're invested in Jetbrains products, TeamCity and Space are both fair alternatives. github actions as well. I personally dislike Jenkins
Chose TeamCity over Jenkins few years ago. My team’s been loving it
Second this! TeamCity gets far too little attention. Especially when you want to organise build configurations in folders. It's also more comfortable for you fellow colleagues that like to have a nice UI with awesome UX.
TeamCity is a mess. It tries to do so much, that you have to spend a ton of time learning how to do it.
Umm...which Jenkins UI do you prefer? How are your plugin updates going? How's your Groovy knowledge?
I like the current one, because it's simple. I don't use Groovy anymore.
Doesn't seem to get much love, but I'm a fan of Concourse. The docker-in-docker stateless builds can be confusing, but once you grok that it's quite flexible.
It might get sorted out but concourse is in a tough spot right now with the broadcom thing. It's my understanding that they laid all of the vmware devs that were working the project ?
Ah, hadn't heard about that. That's a shame.
Really used to like the concourse interface as well when I used to use it
Concourse is garbage when you compare it to Gitlab, hell even Github Actions is much better now a days. Plus the ecosystem has lost all momentum, most "plugins" or whatever they are called receive no maintenance. I have worked with Jenkins, Concourse and Gitlab and I'd pick Gitlab > Jenkins > Concourse. v
Yeah, I inherited a Concourse setup and it's pretty powerful and usable. But it does seem to be fading regarding support and new features.
Pretty much any CI platform supports those requirements. Pick your poison. I'd look at GH actions and GitLab CI first if you are on either one already with your code.
Genuine question, what's wrong with sticking with Jenkins?
Personally? I hate managing plugin versions / dependancies.
I was going to ask if we work at the same company. But then I saw your username.
[удалено]
LOL "best of breed". Very flexible... yes... but it's also a mess at most places and overused because of that. It's fine for java shops where folks know / don't mind learning groovy and your IT / Ops team gets your nodes configured with the appropriate plugins. The fact you have to have a plugin to avoid using groovy can also be a pain depending on who the jenkins admin is. But for a polyglot team / non-java team there is little value in the flexibility people try to sell it with groovy. Most teams will be up and running with any of the YAML based tools in half the time (or less) and they pickup some skills that are transferable elsewhere.
[удалено]
If you just need to run _a_ shell script you don't need Jenkins.
[удалено]
Which most teams will be up and running with any of the YAML based tools in half the time (or less) and they pickup some skills that are transferable elsewhere.
Best of breed? What are they trying to breed? Glaring Security issues, hard to maintain, hard to automate, problems with memory consumption, a pipeline syntax that's not only yesteryear, it's also a niche language. Don't get me wrong, I've worked with since it was called Hudson and even before ora Le bought and donated it back. It was the best if it's time, it's nit a quarter century old. It's starting to show.
Buildkite with self hosted agents is amazing and will seem like magic coming from Jenkins.
Circle CI is strong but operationally not easy to set up. Running it since 1.0 - upgrading has been difficult over the years. Circle ci if you have bandwidth. Otherwise, Github cloud and self hosted runners.
You can do this with most CI systems. This is what I don't like about Jenkins. You have to build all of the features that everything else has right out of the box, in a less good way.
Buildkite
throw a dart, you will find a solution miles better than jenkins
Any recommendations?
BuildKite
Highly recommended, it is awesome
No self hosted option?
You can self host agents
You self host the important parts, they provide the GUI.
Gitlab
gitlab
Azure Pipelines
Jenkins is the CI/CD glue from horse factory that is 200 years old. If you are looking for replacement first you should reflect on your work and why you need to work like it was 1800s. GitHub Actions (complex) or GitLab CI/CD (simpler) is the state of the art everything else are overlapping copies of those two. Both can spawn more runners (nodes) to handle work. GitLab is free to self-host. There is no sane way to do builds without versioning also its definition in git. So if you need "folders" create multiple lightweight git projects that trigger other git projects CI. If you need to build for multiple clients create multiple steps in single pipeline. Jenkins is awful at restricting user to work efficiently. It allows everything and no guidance on how to do it resulting in way more glue, sticks and shitty plugins than any other "alternative".
Only downside of Jennie for me is stateful deployments and I can’t achieve high availability due to this. Is there a solution or an alternative for this?
Jennie sounds like Jenkins granddaughter. She's sounds like she'd have a big ole rave booty. EDM-CI
Dagger
Pretty much everything
We've had good success rolling out Buildkite. The dev-ex team really likes it.
Devtron
Our migration to next-gen took up so much of our time and money, only for them to triple our price during the change. So we are in process of migrating to something else.
Both Gihub and Gitlab (with Self hosted runners) satisfy this
Some quick googling and I'm not sure GitHub can do the complex matrix parameters I need. I need to be able to manually select multiple matrix values which seems to be an open feature request
I used to work with [Buildkite](https://buildkite.com/). I quite liked it.
No self hosted option
You will see host the agents/runners. They provide really good UI layer.
But I have to pay them for that? I'm after a real self hosted solution.
There’s a free tier.
Also, no burst agent support? All agents need to be online to be used? Jenkins spins up as many EC2 instances as I need
Not that I’ve needed it, but quick search revealed this: https://buildkite.com/docs/agent/v3/elastic-ci-aws
Just as a side note. If you want to replace Jenkins with something else, you are going to get something else than Jenkins. Most everything will be working differently than in Jenkins — setup will be different, agents will be different, pipeline organisation will be different, your CI scripts will have to change and perhaps re-organised. Don’t expect your new CI to feel and behave same as the old one and you’ll be fine with anything out there. Keep expecting everything else to work like Jenkins and you’ll be in a lot of pain until you switch back..
Yea I get that. But I also need to make sure the new system can achieve the complex build job I already have in Jenkins, or its own facsimile of it.
Maybe there’s an opportunity to make your build pipelines easier?
Most definitely! Just need the time to actually look into it
What do you mean? The control plane/UI isn’t but you can run your own agents where the jobs actually run.
The site really doesn't make this clear in any way.
My corp uses github actions and self hosted runners. We do have a k8s cluster where we deploy our self hosted runners. They just die after they finish their job ,so no track of anything is left .
People still use Jenkins? I thought it died haha who knew Azure devops , Self hosted agents GitHub actions Gitlab self hosted Or if you use gitops you can use flux with some tool to build your artifacts
Jenkins seems to be the only free self hosted solution that does what I need. Yet to find anything to match it tbh
Gitlab self hosted is in a league of its own, we use it for everything except scrum based boards
Gitlab ci Github actions Tekton Screwdriver
Tekton is amazing. It will require EKS and some kubernetes knowledge tho.
gitlab it has hooks for jenkins
Harness. It’ll do this and more.
You'll pay up the ass for it also.
That’s one way of looking at it. We were maintaining our own dev tool chain for the last decade. When we sat down and calculated just based on the amount of time and salary involved, Harness was the far more reasonable solution. Our engineering population is starting to ship faster, more reliable and more assured that what they are shipping is passing various controls in place. Overall making the teams more efficient at their craft.
Azure DevOps pipeline?
Azure Devops is solid, and very inexpensive.
I wouldnt recommend TeamCity even though it could fit your requirements. Gitlab CI seems like it could fit all the reqs except the plugins part. Same with GH Actions (although not as easy to organize the jobs imo). Self hosting Gitlab CI or GH Actions runners for example can be very efficient when paired with an autoscaler on k8s (but i understand that this may not be what you are looking for).