Enabling Best-Practice Platform Engineering at Lunar
October 18, 2023
Oct 18, 2023
None
Case Study
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Daggernaut Kasper Hermansen (aka @kjuulh on Discord). Kasper is a Platform Engineer at Lunar, a digital bank based in Denmark. He is also the author and maintainer of the Dagger Rust SDK. By switching to Dagger, Kasper and his team were able to apply engineering best practices to Lunar's platform tooling, improve its reliability, and deliver a better experience for Lunar's internal software development teams.
"The most important part about Dagger is that it enables engineering principles across our CI/CD. With Dagger, I can actually trust our roll-out of code." - Kasper Hermansen, Platform Engineer, Lunar
The problem: An unreliable software delivery platform built on shell scripts and templates
Kasper is part of a team of 3 engineers in the DevEx/Tooling squad at Lunar. This small team serves more than 120 developers across the company, providing and improving general purpose developer tooling across the company. The goal of the team is to improve the developer experience by delivering a platform and related tooling for more efficient, consistent, and reliable software engineering.
Before Dagger, Lunar primarily relied on shell scripts and a single Dockerfile with templating. As requirements increased, the Dockerfile ballooned in complexity, and became increasingly difficult to maintain and test. The behavior of the shell scripts was inconsistent, resulting in unexpected results when running in environments that didn't have the expected flags or worked in subtly different ways. The overall reliability of the build system was low, resulting in loss of developer productivity.
Kasper had previously evaluated Dagger in early 2022 in its CUE-focused avatar, but could not find enough compatibility points at the time. In late 2022, Dagger's evolution to a multi-language model, and especially the introduction of the Go SDK, made it a much better fit for Lunar's needs. The open source aspect of Dagger was also important, as it allowed for easy vetting of the source code and the company behind Dagger. After evaluation, the team decided to move forward and trial Dagger as a more sophisticated and reliable alternative to shell scripting.
"We wanted to definitely get out of shell scripts and Docker, because we have to deliver software for a lot of people. It's fine to have a single Dockerfile, but now you're templating the Dockerfile, now you have if statements in that Dockerfile, and now you have 450 lines of Dockerfile to build something that should have been quite simple. So that's why we started the process of moving to Dagger."
The solution: Transitioning the platform to Dagger, one tool at a time
Kasper and his team are currently replacing small parts of the Lunar platform with Dagger:
Lunar uses shuttle, an open source tool, to expose various services to the development team in their local environments. Some of these services are now implemented in Dagger.
For example, developers can use a command like "shuttle run build" on their laptops to create and test a build. This command triggers a Dagger CI pipeline (written using Dagger's Go SDK) that performs all the required tasks locally. This Dagger pipeline also integrates with external services like Snyk and Semgrep for additional code security and analysis functions.
When it's time to create a production image, a similar command and pipeline runs in Lunar's CI system, performing the same tasks and eventually deploying to a self-managed Kubernetes cluster using GitOps. This results in greater consistency between local and remote builds.
As part of their transition from batch processing to real-time streaming, the platform team has developed an internal schema generation tool. This tool requires them to load various files to generate output artifacts. With Dagger, they can do this easily using native Go programming constructs such as loops and filesystem read/write functions.
"I normally evaluate a bunch of tools each month, I test a lot of things all the time. So I'm very quick to adopt something but also discard it again because it doesn't live up to expectations. But for the most part, I've actually been super impressed with the API of Dagger. It just makes sense to me."
The results: Happier developers, more efficient software delivery
Even though the transition to Dagger is still in progress, the platform and development teams are already seeing benefits:
Reliability: The overall reliability of the platform has improved after replacing unstable shell scripts and complex templating with workflows encoded in Go. Developers spend less time debugging unexpected environment-related failures, and this has led to an improvement in productivity.
Consistency: The CI pipeline works the same way for local and remote builds. As a result, errors in the build process are caught early and quickly and there are fewer surprises for developers when pushing final code.
Collaboration: Lunar primarily uses Go for development. Dagger's native Go support makes it easy for developers across the company to inspect platform tools and collaborate with the platform engineering team to improve them. Working entirely in Go also makes it easier for the platform team to follow best practices for testing, packaging and delivering software internally.
"With Dagger, we can test our tooling, we can package it, we can even test it on a test repository if we want to. We can get all this done before it goes out of the door, which is much more difficult with a traditional platform."
The future: Ephemeral, on-demand local development environments and extensions
Although the overall reliability of Lunar's platform has increased after switching to Dagger, the team still sees a fair amount of build failures due to external factors. Kasper's next priority is to add internal telemetry to the Dagger pipeline and start reviewing and analyzing metrics, to gain better visibility into these failures and eventually eliminate them.
Apart from this, Kasper is eagerly awaiting support for extensions in Dagger and is continuing to experiment with ephemeral development environments. His goal with this project is to deliver a set of centrally-managed binaries, which internally use Dagger to create declarative, isolated and reproducible environments on developer laptops.
"We saw Dagger as a CI replacement. But once I began to know it a little bit more and I began to use it a little bit more, I actually saw that you could do so much more with Dagger. There are just so many other uses of Dagger that are super interesting as well."
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Daggernaut Kasper Hermansen (aka @kjuulh on Discord). Kasper is a Platform Engineer at Lunar, a digital bank based in Denmark. He is also the author and maintainer of the Dagger Rust SDK. By switching to Dagger, Kasper and his team were able to apply engineering best practices to Lunar's platform tooling, improve its reliability, and deliver a better experience for Lunar's internal software development teams.
"The most important part about Dagger is that it enables engineering principles across our CI/CD. With Dagger, I can actually trust our roll-out of code." - Kasper Hermansen, Platform Engineer, Lunar
The problem: An unreliable software delivery platform built on shell scripts and templates
Kasper is part of a team of 3 engineers in the DevEx/Tooling squad at Lunar. This small team serves more than 120 developers across the company, providing and improving general purpose developer tooling across the company. The goal of the team is to improve the developer experience by delivering a platform and related tooling for more efficient, consistent, and reliable software engineering.
Before Dagger, Lunar primarily relied on shell scripts and a single Dockerfile with templating. As requirements increased, the Dockerfile ballooned in complexity, and became increasingly difficult to maintain and test. The behavior of the shell scripts was inconsistent, resulting in unexpected results when running in environments that didn't have the expected flags or worked in subtly different ways. The overall reliability of the build system was low, resulting in loss of developer productivity.
Kasper had previously evaluated Dagger in early 2022 in its CUE-focused avatar, but could not find enough compatibility points at the time. In late 2022, Dagger's evolution to a multi-language model, and especially the introduction of the Go SDK, made it a much better fit for Lunar's needs. The open source aspect of Dagger was also important, as it allowed for easy vetting of the source code and the company behind Dagger. After evaluation, the team decided to move forward and trial Dagger as a more sophisticated and reliable alternative to shell scripting.
"We wanted to definitely get out of shell scripts and Docker, because we have to deliver software for a lot of people. It's fine to have a single Dockerfile, but now you're templating the Dockerfile, now you have if statements in that Dockerfile, and now you have 450 lines of Dockerfile to build something that should have been quite simple. So that's why we started the process of moving to Dagger."
The solution: Transitioning the platform to Dagger, one tool at a time
Kasper and his team are currently replacing small parts of the Lunar platform with Dagger:
Lunar uses shuttle, an open source tool, to expose various services to the development team in their local environments. Some of these services are now implemented in Dagger.
For example, developers can use a command like "shuttle run build" on their laptops to create and test a build. This command triggers a Dagger CI pipeline (written using Dagger's Go SDK) that performs all the required tasks locally. This Dagger pipeline also integrates with external services like Snyk and Semgrep for additional code security and analysis functions.
When it's time to create a production image, a similar command and pipeline runs in Lunar's CI system, performing the same tasks and eventually deploying to a self-managed Kubernetes cluster using GitOps. This results in greater consistency between local and remote builds.
As part of their transition from batch processing to real-time streaming, the platform team has developed an internal schema generation tool. This tool requires them to load various files to generate output artifacts. With Dagger, they can do this easily using native Go programming constructs such as loops and filesystem read/write functions.
"I normally evaluate a bunch of tools each month, I test a lot of things all the time. So I'm very quick to adopt something but also discard it again because it doesn't live up to expectations. But for the most part, I've actually been super impressed with the API of Dagger. It just makes sense to me."
The results: Happier developers, more efficient software delivery
Even though the transition to Dagger is still in progress, the platform and development teams are already seeing benefits:
Reliability: The overall reliability of the platform has improved after replacing unstable shell scripts and complex templating with workflows encoded in Go. Developers spend less time debugging unexpected environment-related failures, and this has led to an improvement in productivity.
Consistency: The CI pipeline works the same way for local and remote builds. As a result, errors in the build process are caught early and quickly and there are fewer surprises for developers when pushing final code.
Collaboration: Lunar primarily uses Go for development. Dagger's native Go support makes it easy for developers across the company to inspect platform tools and collaborate with the platform engineering team to improve them. Working entirely in Go also makes it easier for the platform team to follow best practices for testing, packaging and delivering software internally.
"With Dagger, we can test our tooling, we can package it, we can even test it on a test repository if we want to. We can get all this done before it goes out of the door, which is much more difficult with a traditional platform."
The future: Ephemeral, on-demand local development environments and extensions
Although the overall reliability of Lunar's platform has increased after switching to Dagger, the team still sees a fair amount of build failures due to external factors. Kasper's next priority is to add internal telemetry to the Dagger pipeline and start reviewing and analyzing metrics, to gain better visibility into these failures and eventually eliminate them.
Apart from this, Kasper is eagerly awaiting support for extensions in Dagger and is continuing to experiment with ephemeral development environments. His goal with this project is to deliver a set of centrally-managed binaries, which internally use Dagger to create declarative, isolated and reproducible environments on developer laptops.
"We saw Dagger as a CI replacement. But once I began to know it a little bit more and I began to use it a little bit more, I actually saw that you could do so much more with Dagger. There are just so many other uses of Dagger that are super interesting as well."
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Daggernaut Kasper Hermansen (aka @kjuulh on Discord). Kasper is a Platform Engineer at Lunar, a digital bank based in Denmark. He is also the author and maintainer of the Dagger Rust SDK. By switching to Dagger, Kasper and his team were able to apply engineering best practices to Lunar's platform tooling, improve its reliability, and deliver a better experience for Lunar's internal software development teams.
"The most important part about Dagger is that it enables engineering principles across our CI/CD. With Dagger, I can actually trust our roll-out of code." - Kasper Hermansen, Platform Engineer, Lunar
The problem: An unreliable software delivery platform built on shell scripts and templates
Kasper is part of a team of 3 engineers in the DevEx/Tooling squad at Lunar. This small team serves more than 120 developers across the company, providing and improving general purpose developer tooling across the company. The goal of the team is to improve the developer experience by delivering a platform and related tooling for more efficient, consistent, and reliable software engineering.
Before Dagger, Lunar primarily relied on shell scripts and a single Dockerfile with templating. As requirements increased, the Dockerfile ballooned in complexity, and became increasingly difficult to maintain and test. The behavior of the shell scripts was inconsistent, resulting in unexpected results when running in environments that didn't have the expected flags or worked in subtly different ways. The overall reliability of the build system was low, resulting in loss of developer productivity.
Kasper had previously evaluated Dagger in early 2022 in its CUE-focused avatar, but could not find enough compatibility points at the time. In late 2022, Dagger's evolution to a multi-language model, and especially the introduction of the Go SDK, made it a much better fit for Lunar's needs. The open source aspect of Dagger was also important, as it allowed for easy vetting of the source code and the company behind Dagger. After evaluation, the team decided to move forward and trial Dagger as a more sophisticated and reliable alternative to shell scripting.
"We wanted to definitely get out of shell scripts and Docker, because we have to deliver software for a lot of people. It's fine to have a single Dockerfile, but now you're templating the Dockerfile, now you have if statements in that Dockerfile, and now you have 450 lines of Dockerfile to build something that should have been quite simple. So that's why we started the process of moving to Dagger."
The solution: Transitioning the platform to Dagger, one tool at a time
Kasper and his team are currently replacing small parts of the Lunar platform with Dagger:
Lunar uses shuttle, an open source tool, to expose various services to the development team in their local environments. Some of these services are now implemented in Dagger.
For example, developers can use a command like "shuttle run build" on their laptops to create and test a build. This command triggers a Dagger CI pipeline (written using Dagger's Go SDK) that performs all the required tasks locally. This Dagger pipeline also integrates with external services like Snyk and Semgrep for additional code security and analysis functions.
When it's time to create a production image, a similar command and pipeline runs in Lunar's CI system, performing the same tasks and eventually deploying to a self-managed Kubernetes cluster using GitOps. This results in greater consistency between local and remote builds.
As part of their transition from batch processing to real-time streaming, the platform team has developed an internal schema generation tool. This tool requires them to load various files to generate output artifacts. With Dagger, they can do this easily using native Go programming constructs such as loops and filesystem read/write functions.
"I normally evaluate a bunch of tools each month, I test a lot of things all the time. So I'm very quick to adopt something but also discard it again because it doesn't live up to expectations. But for the most part, I've actually been super impressed with the API of Dagger. It just makes sense to me."
The results: Happier developers, more efficient software delivery
Even though the transition to Dagger is still in progress, the platform and development teams are already seeing benefits:
Reliability: The overall reliability of the platform has improved after replacing unstable shell scripts and complex templating with workflows encoded in Go. Developers spend less time debugging unexpected environment-related failures, and this has led to an improvement in productivity.
Consistency: The CI pipeline works the same way for local and remote builds. As a result, errors in the build process are caught early and quickly and there are fewer surprises for developers when pushing final code.
Collaboration: Lunar primarily uses Go for development. Dagger's native Go support makes it easy for developers across the company to inspect platform tools and collaborate with the platform engineering team to improve them. Working entirely in Go also makes it easier for the platform team to follow best practices for testing, packaging and delivering software internally.
"With Dagger, we can test our tooling, we can package it, we can even test it on a test repository if we want to. We can get all this done before it goes out of the door, which is much more difficult with a traditional platform."
The future: Ephemeral, on-demand local development environments and extensions
Although the overall reliability of Lunar's platform has increased after switching to Dagger, the team still sees a fair amount of build failures due to external factors. Kasper's next priority is to add internal telemetry to the Dagger pipeline and start reviewing and analyzing metrics, to gain better visibility into these failures and eventually eliminate them.
Apart from this, Kasper is eagerly awaiting support for extensions in Dagger and is continuing to experiment with ephemeral development environments. His goal with this project is to deliver a set of centrally-managed binaries, which internally use Dagger to create declarative, isolated and reproducible environments on developer laptops.
"We saw Dagger as a CI replacement. But once I began to know it a little bit more and I began to use it a little bit more, I actually saw that you could do so much more with Dagger. There are just so many other uses of Dagger that are super interesting as well."
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!
