Restoring Developer Confidence in CI

We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Daggernaut Sean Clayton (aka @clayton9821 on Discord). Sean recently joined a new company, and was surprised to learn he would be the SME for all its CI pipelines. He had never used GitLab's CI/CD feature before, but took initiative to make it better. By switching to Dagger and away from a YAML-based CI workflow, Sean was able to give the development team faster builds and increased.

"How many times have you pushed your code to the pipeline that works locally and then it fails for some reason in the remote pipeline and you have no idea why? Dagger solves that. By letting you see your pipeline as code, you can do everything that you can in production locally, and then push to the pipeline with so much more confidence."
- Sean Clayton

The problem: Team frustration with unreliable CI

Before Dagger, the existing CI pipeline consisted of 1000+ lines of YAML code, with both actions and templates mixed together. It was so complex that no one in the team understood it completely. Due to the high degree of complexity, the pipeline code was difficult to maintain and update, and CI failures could not be debugged and resolved quickly. These issues were increasing developer friction and frustration, and affecting the team's confidence in the entire CI workflow.

Sean was keen to improve the experience for the development team and started investigating alternatives. During the process, he attended a talk by the Dagger team at KubeHuddle in Toronto and immediately realized that Dagger provided the modern and maintainable approach he was searching for. He started by reading the Dagger documentation, built a proof-of-concept and then began the process of migrating the existing CI pipeline to Dagger.

"When the pipeline failed, people would come to the SRE team and say, it must be the pipeline that's broken. And that was really what drove me to want to improve this, rather than just keep applying band-aids over and over".

The solution: Faster feedback loops and more consistent CI pipelines

With Dagger, Sean has been able to both simplify the CI pipeline and make it more consistent. The Dagger pipeline performs the same steps as the previous YAML pipeline, but using Dagger's Python SDK (for example, replacing a Packer template with Python SDK methods). It operates in a blended environment: QA on Kubernetes and production on bare metal.

1. The pipeline begins by building and tagging a Docker image of the application.

2. It uses Terragrunt to access the Kubernetes cluster and configures traffic mirroring rules with Istio.

3. It builds and tags a build RPM and deploys the build to the Kubernetes QA environment.

4. It builds and tags a production RPM.

5. It deploys the production RPM to the production environment on bare metal.

"Local testing was the big win because the biggest thing we heard is, the build works locally and it's not working through the pipeline. Having the developers be able to push to the new pre-QA environment and see how the pipeline works was huge."

Improved local loop: The same Dagger pipeline works consistently locally and remotely. As a result, developers are able to build and test locally, exactly as they would through GitLab. Instead of pushing code to GitLab and waiting to see if builds succeed, they now receive almost instantaneous local feedback on their builds.

• Quicker builds: Dagger's support for cache volumes produces significantly faster builds. Previously, a high percentage of build time went in downloading Maven packages. By using cache volumes to store Maven downloads, Dagger has reduced the time required for each Maven build, which in turn reduces the time taken for a new feature to go from development to production.

• Higher efficiency and confidence levels: The CI pipeline is now encoded in Python; there is no longer any lengthy and complex YAML configuration to deal with. This has made the pipeline easier to understand, debug, and extend, thus enabling the SRE team to work more efficiently. Since developers can also inspect and understand the pipeline code directly, their confidence levels in the CI process are also higher.

"It's definitely faster, and the developer that I'm working with on this is extremely pleased. Giving devs the ability to replicate the pipeline locally increases development speed dramatically. We both know that as soon as we introduce this, it's going to be adopted very quickly."

The future: Building platform-independent images and front-end testing with Dagger

Sean is working on moving more internal customers over to the new Dagger pipeline, and has various ideas for future improvements:

• Sean is currently working with the development team to move from an RPM build pipeline to a cloud-native Java application built with Dagger. Sean is able to do this much faster because of the local Dagger pipeline.

• The Dagger Engine currently runs on a Kubernetes cluster, alongside a self-hosted CI runner. When new Dagger Engine pods spin up, they start with an empty cache. As a result, initial pipeline runs take longer. Sean is planning to create a tool to prime the cache quickly, so that developers don't need to wait for the cache to fill.

• For front-end testing requirements, Sean is investigating how to have Python pipeline code integrate with JavaScript-based tests, either by directly calling the Dagger API using HTTP or by translating some of the Python pipeline code to JavaScript.

"The possibility for innovation with Dagger is so huge. Our pipeline is so much shorter, much faster and can be run on any operating system. This was a win directly because we were using Dagger."

Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!