Dagger 0.5: New Dynamic Secrets API and more secure scrubbing

Dagger allows you to use confidential information, such as passwords, API keys, SSH keys and so on, when running your Dagger pipelines. Keeping these secrets secure while also ensuring they are accessible to the right people is a common challenge.

At Dagger, we understand this challenge and are always looking for ways to make secrets management easier and more secure. That's why we're excited to announce the launch of a new dynamic secrets API in Dagger 0.5.

“Build pipelines and deployments that don’t need secrets are about as useful as deploying to /dev/null. So with the new secrets API available with Dagger, my pipelines have gone from “fun idea” to “oh crap, we’re shipping to prod” and I couldn’t be more excited. Finally I have a programmatic and composable build pipeline I can run anywhere. And I really mean anywhere.”
– David Flanagan, Founder, Rawkode Academy

The challenges of Dynamic Secrets

Before

Until this release, Dagger could securely load secrets from a fixed snapshot of the client machine’s environment variables. This works well for simple workflows, where all secrets are static and known in advance. However, many CI/CD workflows now require retrieving secrets dynamically, for example by querying a secret store such as Hashicorp Vault, executing a cloud-specific authentication helper, or integrating with a password manager. Implementing these workflows with Dagger was only possible with unreliable and possibly insecure hacks - until now.

After

Our new secrets API enables you to retrieve secrets however you like, including from external sources such as Vault, Scaleway, GCP, Azure, AWS, and others (using their pre-defined systems, SDKs or tools).Then, you set them in your Dagger pipeline either as secret environment variables, or as mounted secret files.

Here’s a quick example, which reads a plaintext GitHub personal access token from a host environment variable named GH_SECRET and sets it as a secret for a Dagger pipeline:

package main

import (
   "context"
   "fmt"
   "os"

   "dagger.io/dagger"
)

func main() {
   // initialize Dagger client
   ctx := context.Background()
   client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
   if err != nil {
       panic(err)
   }
   defer client.Close()

   // read secret from host variable
   secret := client.SetSecret("gh-secret", os.Getenv("GH_SECRET"))

   // use secret in container environment
   out, err := client.
       Container().
       From("alpine:3.17").
       WithSecretVariable("GITHUB_API_TOKEN", secret).
       WithExec([]string{"apk", "add", "curl"}).
       WithExec([]string{"sh", "-c", `curl "https://api.github.com/repos/dagger/dagger/issues" --header "Accept: application/vnd.github+json" --header "Authorization: Bearer $GITHUB_API_TOKEN"`}).
       Stdout(ctx)
   if err != nil {
       panic(err)
   }

   fmt.Println(out)
}

Notes:

• File secrets are mounted on a special mount, and if mounted elsewhere, they are not part of the image, nor the cache. This is the most secure option.

• Environment secrets are not part of the build environment (like build-args can be), either, so they don't leak in the build history (you can't access them using docker history, for example).

Security

As of Dagger 0.4, Dagger scrubs secrets from its logs to ensure that sensitive information is protected, and it’s now better than before - our scrubbing strategy can now handle multi-line secrets such as SSH keys or TLS certificates. This applies to secrets stored in both environment variables and file mounts.

What's Next?

We have more improvements planned to the secrets API. For example, we're working on improving our scrubbing strategy to further reduce the risk of leaks. We're also exploring ways to give you more control over secret mounts, such as UID/GID and Linux filesystem rights. If you're using a Dockerfile and want to pass a secret during the build, we don't support that yet, but have an issue open to fix it.

To learn more about our new secrets API, refer to our documentation, which includes several code samples to get you started quickly. If you have feedback, please share your comments with us in Discord or create a GitHub issue.

Want to help us pick the next improvements to the secrets API? Vote on our latest discussion post.