Cutting Deployment Times from 3 Hours to 3 Minutes
March 16, 2023
Mar 16, 2023
None
Case Study
“Replacing our makefiles with Dagger has greatly simplified our process for deployments since it’s much faster (1-3 hours to 3 minutes)...The power, flexibility and capability behind Dagger is something that gets me excited.”
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Dagger community member, Andy Gonzalez (aka @duchi on Discord). Andy, an SRE team leader, used Dagger to speed up his production AWS Lambda deployments from a few hours to just a few minutes…a 93% reduction in processing time!
The problem: Manual, time-consuming deployments and complex YAML
Andy and his team deploy their application using a CircleCI pipeline. This pipeline is a 3000+-line YAML file that performs multiple tasks for their core application. Besides those core tasks, there is a need to deploy 15 different AWS Lambdas (serverless functions) to support the primary application across multiple environments and using different accounts. To begin with, this was a totally manual process, and incorporating it into the existing CI YAML was daunting. Nevertheless, something had to be done since the manual Lambda deployments took between 1 and 3 hours every time!
Each Lambda is unique – it performs a different task and has its own set of dependencies. Some examples include antivirus scanning of file uploads by customers, RDS snapshot backups in multiple regions, client side logging and more. These functions are updated and re-deployed once per week.
There were a few challenges with this approach:
All the Terraform commands to deploy the Lambdas were defined in Makefiles. As the number of environments increased, these Makefiles started to become increasingly complex and hard to maintain.
The entire YAML pipeline – all 3000+ lines of it – was heavily tied to CircleCI. This effectively locked the team into a specific platform and made it difficult to consider shifting to a different CI vendor in future.
The Lambda deployment process had zero automation in place; it was manual and extremely time-consuming. Typically, it would take between 1-3 hours.
“Before we started to use Dagger, we deployed our Lambdas one at a time between all of our environments. We have four accounts with each containing one or more ‘environments’, so this would take a long time. Here is what our old manual steps would look like: (1) Get credentials and specify environment; (2) Deploy or update Lambda using Terraform CLI; (3) Verify Lambdas are behaving correctly; (4) Move on to the next environment.”
The solution: Fast, parallelized deployment with Dagger and a matrix
Andy had heard about Dagger and, as an initial proof-of-concept, he decided to try and automate the AWS Lambda deployment sub-process using Dagger. With Dagger, he was able to consolidate all the tasks related to Lambda deployment in a separate Dagger pipeline and run them alongside matrix jobs in CircleCI.
Under the new approach, the AWS Lambda deployment process is handled in a separate Dagger pipeline outside the CircleCI pipeline. When triggered, this pipeline performs the following tasks:
It creates a Terraform plan and puts it in the comment of a PR.
The Terraform plan is executed manually for each Lambda.
Control then passes to CircleCI, which executes the Dagger pipeline for each environment using a matrix job.
Key benefits of this new approach are:
Using a matrix job allows the Dagger pipelines for the different environments to run in parallel, significantly speeding up the deployment process. Lambdas are now deployed in roughly 3-4 minutes instead of the 1-3 hours required earlier.
Deployment business logic is encapsulated in the Dagger pipelines, with CircleCI only providing the matrix job infrastructure and triggering the pipelines. This reduces the dependency on CircleCI and makes it easier to migrate to a different platform in future.
Testing the pipeline logic is now much easier, as pipelines can be run locally. Deployments can also be performed either via local runs or from CircleCI.
“Dagger has allowed our engineers to deploy at a higher velocity and focus on other items instead of the manual workflow of deploying Lambdas. Our new Dagger process looks like this: (1) Run Dagger pipeline in CircleCI; (2) Verify Lambdas are behaving correctly.”
The future: Eliminating YAML and improving secrets management
Going forward, Andy plans to combine the separate Lambda deployment pipeline with the existing CircleCI pipeline and simultaneously begin migrating the CircleCI pipeline to Dagger. He expects this will make the deployment process easier to test and maintain, and less reliant on the CircleCI platform.
In the CircleCI matrix, each job currently uses OIDC to connect to AWS and pass credentials to Dagger via environment variables. Andy will be looking for a better way to pass secrets through to executions – for example, using Dagger to connect to AWS directly for secrets.
“Dagger is an extremely powerful tool which allows users to upshift from any CI/CD platform without hesitation. I believe that Dagger is the future for CI/CD since the tool allows users to be so decoupled by the actual platform itself.”
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!
“Replacing our makefiles with Dagger has greatly simplified our process for deployments since it’s much faster (1-3 hours to 3 minutes)...The power, flexibility and capability behind Dagger is something that gets me excited.”
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Dagger community member, Andy Gonzalez (aka @duchi on Discord). Andy, an SRE team leader, used Dagger to speed up his production AWS Lambda deployments from a few hours to just a few minutes…a 93% reduction in processing time!
The problem: Manual, time-consuming deployments and complex YAML
Andy and his team deploy their application using a CircleCI pipeline. This pipeline is a 3000+-line YAML file that performs multiple tasks for their core application. Besides those core tasks, there is a need to deploy 15 different AWS Lambdas (serverless functions) to support the primary application across multiple environments and using different accounts. To begin with, this was a totally manual process, and incorporating it into the existing CI YAML was daunting. Nevertheless, something had to be done since the manual Lambda deployments took between 1 and 3 hours every time!
Each Lambda is unique – it performs a different task and has its own set of dependencies. Some examples include antivirus scanning of file uploads by customers, RDS snapshot backups in multiple regions, client side logging and more. These functions are updated and re-deployed once per week.
There were a few challenges with this approach:
All the Terraform commands to deploy the Lambdas were defined in Makefiles. As the number of environments increased, these Makefiles started to become increasingly complex and hard to maintain.
The entire YAML pipeline – all 3000+ lines of it – was heavily tied to CircleCI. This effectively locked the team into a specific platform and made it difficult to consider shifting to a different CI vendor in future.
The Lambda deployment process had zero automation in place; it was manual and extremely time-consuming. Typically, it would take between 1-3 hours.
“Before we started to use Dagger, we deployed our Lambdas one at a time between all of our environments. We have four accounts with each containing one or more ‘environments’, so this would take a long time. Here is what our old manual steps would look like: (1) Get credentials and specify environment; (2) Deploy or update Lambda using Terraform CLI; (3) Verify Lambdas are behaving correctly; (4) Move on to the next environment.”
The solution: Fast, parallelized deployment with Dagger and a matrix
Andy had heard about Dagger and, as an initial proof-of-concept, he decided to try and automate the AWS Lambda deployment sub-process using Dagger. With Dagger, he was able to consolidate all the tasks related to Lambda deployment in a separate Dagger pipeline and run them alongside matrix jobs in CircleCI.
Under the new approach, the AWS Lambda deployment process is handled in a separate Dagger pipeline outside the CircleCI pipeline. When triggered, this pipeline performs the following tasks:
It creates a Terraform plan and puts it in the comment of a PR.
The Terraform plan is executed manually for each Lambda.
Control then passes to CircleCI, which executes the Dagger pipeline for each environment using a matrix job.
Key benefits of this new approach are:
Using a matrix job allows the Dagger pipelines for the different environments to run in parallel, significantly speeding up the deployment process. Lambdas are now deployed in roughly 3-4 minutes instead of the 1-3 hours required earlier.
Deployment business logic is encapsulated in the Dagger pipelines, with CircleCI only providing the matrix job infrastructure and triggering the pipelines. This reduces the dependency on CircleCI and makes it easier to migrate to a different platform in future.
Testing the pipeline logic is now much easier, as pipelines can be run locally. Deployments can also be performed either via local runs or from CircleCI.
“Dagger has allowed our engineers to deploy at a higher velocity and focus on other items instead of the manual workflow of deploying Lambdas. Our new Dagger process looks like this: (1) Run Dagger pipeline in CircleCI; (2) Verify Lambdas are behaving correctly.”
The future: Eliminating YAML and improving secrets management
Going forward, Andy plans to combine the separate Lambda deployment pipeline with the existing CircleCI pipeline and simultaneously begin migrating the CircleCI pipeline to Dagger. He expects this will make the deployment process easier to test and maintain, and less reliant on the CircleCI platform.
In the CircleCI matrix, each job currently uses OIDC to connect to AWS and pass credentials to Dagger via environment variables. Andy will be looking for a better way to pass secrets through to executions – for example, using Dagger to connect to AWS directly for secrets.
“Dagger is an extremely powerful tool which allows users to upshift from any CI/CD platform without hesitation. I believe that Dagger is the future for CI/CD since the tool allows users to be so decoupled by the actual platform itself.”
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!
“Replacing our makefiles with Dagger has greatly simplified our process for deployments since it’s much faster (1-3 hours to 3 minutes)...The power, flexibility and capability behind Dagger is something that gets me excited.”
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Dagger community member, Andy Gonzalez (aka @duchi on Discord). Andy, an SRE team leader, used Dagger to speed up his production AWS Lambda deployments from a few hours to just a few minutes…a 93% reduction in processing time!
The problem: Manual, time-consuming deployments and complex YAML
Andy and his team deploy their application using a CircleCI pipeline. This pipeline is a 3000+-line YAML file that performs multiple tasks for their core application. Besides those core tasks, there is a need to deploy 15 different AWS Lambdas (serverless functions) to support the primary application across multiple environments and using different accounts. To begin with, this was a totally manual process, and incorporating it into the existing CI YAML was daunting. Nevertheless, something had to be done since the manual Lambda deployments took between 1 and 3 hours every time!
Each Lambda is unique – it performs a different task and has its own set of dependencies. Some examples include antivirus scanning of file uploads by customers, RDS snapshot backups in multiple regions, client side logging and more. These functions are updated and re-deployed once per week.
There were a few challenges with this approach:
All the Terraform commands to deploy the Lambdas were defined in Makefiles. As the number of environments increased, these Makefiles started to become increasingly complex and hard to maintain.
The entire YAML pipeline – all 3000+ lines of it – was heavily tied to CircleCI. This effectively locked the team into a specific platform and made it difficult to consider shifting to a different CI vendor in future.
The Lambda deployment process had zero automation in place; it was manual and extremely time-consuming. Typically, it would take between 1-3 hours.
“Before we started to use Dagger, we deployed our Lambdas one at a time between all of our environments. We have four accounts with each containing one or more ‘environments’, so this would take a long time. Here is what our old manual steps would look like: (1) Get credentials and specify environment; (2) Deploy or update Lambda using Terraform CLI; (3) Verify Lambdas are behaving correctly; (4) Move on to the next environment.”
The solution: Fast, parallelized deployment with Dagger and a matrix
Andy had heard about Dagger and, as an initial proof-of-concept, he decided to try and automate the AWS Lambda deployment sub-process using Dagger. With Dagger, he was able to consolidate all the tasks related to Lambda deployment in a separate Dagger pipeline and run them alongside matrix jobs in CircleCI.
Under the new approach, the AWS Lambda deployment process is handled in a separate Dagger pipeline outside the CircleCI pipeline. When triggered, this pipeline performs the following tasks:
It creates a Terraform plan and puts it in the comment of a PR.
The Terraform plan is executed manually for each Lambda.
Control then passes to CircleCI, which executes the Dagger pipeline for each environment using a matrix job.
Key benefits of this new approach are:
Using a matrix job allows the Dagger pipelines for the different environments to run in parallel, significantly speeding up the deployment process. Lambdas are now deployed in roughly 3-4 minutes instead of the 1-3 hours required earlier.
Deployment business logic is encapsulated in the Dagger pipelines, with CircleCI only providing the matrix job infrastructure and triggering the pipelines. This reduces the dependency on CircleCI and makes it easier to migrate to a different platform in future.
Testing the pipeline logic is now much easier, as pipelines can be run locally. Deployments can also be performed either via local runs or from CircleCI.
“Dagger has allowed our engineers to deploy at a higher velocity and focus on other items instead of the manual workflow of deploying Lambdas. Our new Dagger process looks like this: (1) Run Dagger pipeline in CircleCI; (2) Verify Lambdas are behaving correctly.”
The future: Eliminating YAML and improving secrets management
Going forward, Andy plans to combine the separate Lambda deployment pipeline with the existing CircleCI pipeline and simultaneously begin migrating the CircleCI pipeline to Dagger. He expects this will make the deployment process easier to test and maintain, and less reliant on the CircleCI platform.
In the CircleCI matrix, each job currently uses OIDC to connect to AWS and pass credentials to Dagger via environment variables. Andy will be looking for a better way to pass secrets through to executions – for example, using Dagger to connect to AWS directly for secrets.
“Dagger is an extremely powerful tool which allows users to upshift from any CI/CD platform without hesitation. I believe that Dagger is the future for CI/CD since the tool allows users to be so decoupled by the actual platform itself.”
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!
