Adopting a Monorepo Strategy: Civo’s Experience

We’re always interested in hearing how our community uses Dagger – their use cases, challenges, and experiences deploying it in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Daggernaut Dinesh Majrekar (aka @DMajrekar on Discord). Dinesh is the CTO of Civo, a cloud-native service provider focused on managed Kubernetes and Machine Learning environments. By using Dagger, Dinesh and his team were able to speed up the CI/CD pipelines in their monorepo and transform the internal developer experience by enabling teams to independently and reliably write and test their own pipelines. The faster build times (down from 30 minutes to 5 minutes) were a happy bonus!

“The greatest thing about Dagger is that it lets you unit test your CI pipeline locally, in the same tools that you use to write code." - Dinesh Majrekar,  CTO of Civo

Migrating Inter-Dependent Repositories and CI Pipelines to a Monorepo

Initially, Civo had more than 40 repositories, all with inter-connected dependencies. CI/CD pipelines and job visibility was handled by GitLab CI. To simplify dependency management and make collaboration easier, Civo decided to adopt a monorepo strategy and, in mid-2023, started the process of moving its projects into a single repository. As part of this effort, the team also began looking at ways to optimize and streamline the CI/CD architecture.

Civo had experience with other CI/CD tools, with implementations of GitLab pipelines, Github Actions and Argo Workflows. However, Dinesh and his team were excited to transition away from YAML for defining their pipelines in favor of the programming language they used internally - Go.

Dagger's open source license was also an important factor in this decision; as a cloud provider, Civo has to meet strict legal and compliance requirements and its internal policies only permit use of truly open source projects.

“With Dagger, the ability to write pipelines in Go means that developers are empowered to manage their own pipelines, rather than handing it over to the SRE team, where Bash and YAML are the order of the day.”

“Daggerizing” the Monorepo: A Smarter, Faster and More Reliable CI Pipeline

The monorepo migration project has been a huge success for Dinesh and his team. The codebase is centralized, internal teams have greater visibility into each other’s work and can collaborate better, and operational friction has decreased. The project has felt like a huge step forward for the Civo team compared to their old way of doing things.

The new monorepo now also uses Dagger for all CI/CD operations. The switch to Dagger has resulted in a number of benefits:

• Smarter builds: Using Dagger gave the Civo team a programmable engine and powerful  caching capabilities, allowing Civo to intelligently build (or not build) parts of the monorepo, depending on what's changed and what needs re-testing. For example, different projects in Civo's monorepo use different Go versions. Civo's Dagger pipeline is able to dynamically extract the Go version for each project and then build and test the codebase using the correct version. This degree of customizability and control was not available in the previous CI/CD system and made a huge difference to the Civo team.

• Improved reliability: In the previous example, the pipeline code that identifies and extracts the Go version is a critical component. Since Civo's Dagger pipelines are written in native Go, it's now easy for the development team to add unit tests to validate all such critical components, using the Go testing tools they're already familiar with. These unit tests, which are also tested within the generic Dagger pipeline, have improved the overall reliability of the pipeline and also opened up opportunities for code reuse.

• Better developer experience: The sub-projects in Civo's monorepo also use Dagger internally. For example, if a project needs an SQL database for its unit tests, a Dagger pipeline spins up a transient database service and attaches it to the test. By shifting the CI/CD pipeline closer to the application codebase, Dinesh and his team have made it easier for internal developers to test code locally, while also reducing their reliance on SREs.

"At peak time, when we're getting ready for a conference and we're in that deploy frenzy, we're doing 30 to 40 deployments every day. The speed boost from using Dagger is very noticeable - it has cut our build time from 30 minutes to 5 minutes."

Future Plans

Dinesh and the Civo team intend to expand their use of Dagger in the future.

• Currently, the Dagger pipelines are all written in Go. However, as more and more projects are migrated to the monorepo, the team plans to create multi-language pipelines using Dagger's Python and Rust SDKs.

• The team has also been following the ongoing work on Dagger modules and is intrigued by the promise of using modules to improve the current pipeline and to speed up Civo's machine learning tools.

"Dagger has just seamlessly worked for us, especially the caching. Our developers are happy that everything is so much faster."

Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!